Method, computer readable storage medium and apparatus for pre-processing winograd convolution

Abstract

The application relates to a method for preprocessing neuron data and weight data of a Winograd convolution, a computer readable storage medium and a combined processing device. The method comprises the following steps: setting the size of a data unit as the product of a vectorization direction and a data bit number; and aligning storage of the neuron data and the weight data in units of the data unit. The application has the technical effects of guaranteeing network precision, accelerating performance, reducing area and reducing power consumption.

Description

Technical Field

[0001] This invention generally relates to the field of neural networks. More specifically, this invention relates to a method for preprocessing neuron data and weight data of Winograd convolutions, a computer-readable storage medium, and a combined processing apparatus. Background Technology

[0002] With the rapid development of the information age, research in the fields of artificial intelligence and machine learning is booming, and related industries are flourishing. Convolutional neural networks have a wide range of applications in computer vision, autonomous driving, machine translation, speech recognition, smart homes, and many other areas.

[0003] The large number of parameters and computational demands of convolutional neural networks severely limit their performance in portable mobile devices with limited area and computing power. Furthermore, non-dedicated processors also incur significant power consumption when performing convolution operations.

[0004] Winograd convolution is a method for accelerating convolution based on multinomial interpolation. It involves dividing the two inputs of the convolution operation—neurons and weights—by a certain scale, performing a linear transformation (Winograd forward transform) on each, then multiplying the transformed neurons and weights in pairs, and finally performing a linear transformation (Winograd inverse transform) on the result. The result is then equivalent to the original convolution operation.

[0005] Since the forward and inverse transformation matrices of neurons and weights in the Winograd convolution operation are composed of simple, fixed values, the forward and inverse transformation processes of Winograd neurons and weights can be implemented using only addition. The multiplication operations required in the Winograd algorithm only occur during positional multiplication, and the multiplication complexity of this process is significantly reduced compared to the original convolution algorithm. Because the hardware overhead (timing, power consumption, area) for implementing multiplication is much higher than implementing addition of the same bit width, replacing the original convolution operation with Winograd convolution brings significant benefits in hardware energy efficiency and computation time.

[0006] However, no hardware is currently designed specifically for accelerating the Winograd convolution algorithm, preventing existing AI chips from fully leveraging the advantages of Winograd convolution operations. Therefore, a solution capable of efficiently preprocessing Winograd convolution algorithms is urgently needed. Summary of the Invention

[0007] To at least partially address the technical problems mentioned in the background art, the present invention provides a method, a computer-readable storage medium, and a combined processing apparatus for preprocessing neuron data and weight data of Winograd convolutions.

[0008] In one aspect, the present invention discloses a method for preprocessing neuron data and weight data of Winograd convolution, comprising: setting the size of the data unit as the product of the vectorization direction and the number of bits of data; and storing the neuron data in aligned units of the data units.

[0009] In another aspect, the present invention discloses a computer-readable storage medium storing computer program code on which preprocessed neuron data and weight data of Winograd convolution are stored, wherein the aforementioned method is executed when the computer program code is run by a processing device.

[0010] In another aspect, the present invention discloses a combined processing apparatus for preprocessing neuron data and weight data from Winograd convolutions, including a processing device and off-chip memory. The processing device is used to set the size of the data unit as the product of the vectorization direction and the number of bits of data, and the off-chip memory is used to store the neuron data in aligned units.

[0011] The preprocessing scheme proposed in this invention can match the Winograd convolution acceleration algorithm, and has the technical effects of ensuring network accuracy, performance acceleration, area reduction and power consumption reduction. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0013] Figure 1 This is a schematic diagram illustrating the convolution operation between the convolution kernel and the input neuron image;

[0014] Figure 2 This is a schematic diagram showing the transformation of the original convolution of F(2×2,3×3) into a Winograd convolution;

[0015] Figure 3 This is a visual diagram illustrating the bitwise multiplication operation;

[0016] Figure 4 This is a schematic diagram illustrating the homogeneous operation between the positively transformed data and the weights;

[0017] Figure 5 This is a structural diagram of the board card according to an embodiment of the present invention;

[0018] Figure 6 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present invention;

[0019] Figure 7 This is a schematic diagram illustrating the preprocessing of neuron data with two Vickers number elements.

[0020] Figure 8 This is a flowchart illustrating the preprocessing of Winograd convolution according to an embodiment of the present invention;

[0021] Figure 9 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present invention;

[0022] Figure 10 This is a schematic diagram showing the overlapping portion during a positive transformation;

[0023] Figure 11 This is a schematic diagram illustrating a neuron cache according to an embodiment of the present invention;

[0024] Figure 12 This is a schematic diagram illustrating the positive transformation unit according to an embodiment of the present invention;

[0025] Figure 13 This is a schematic diagram illustrating the positive transform data buffer according to an embodiment of the present invention;

[0026] Figure 14 This is a schematic diagram illustrating the weight cache of an embodiment of the present invention;

[0027] Figure 15 This is a schematic diagram illustrating the output side of the positive transform data buffer according to an embodiment of the present invention;

[0028] Figure 16 This is a schematic diagram illustrating the weight cache output side of an embodiment of the present invention;

[0029] Figure 17 This is a schematic diagram illustrating the inverse transformation unit of an embodiment of the present invention; and

[0030] Figure 18 This is a schematic diagram illustrating the connection relationship of the result cache in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0033] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0034] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0035] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0036] The Winograd convolution acceleration algorithm (hereinafter referred to as the Winograd algorithm or Winograd convolution) utilizes a linear transformation on the operands in convolution operations to find the transformation method that requires the fewest multiplications, and then replaces the necessary multiplication operations with some addition operations. From a hardware perspective, multipliers are more complex in structure, consume more power, and have worse overall processing performance than adders. In practice, the Winograd algorithm, which replaces multiplication with addition, has a significant advantage in processing convolution operations.

[0037] For two-dimensional convolution, assuming the input neuron image size is H×W (H is the height of the input neuron image, W is the width of the input neuron image), and the weights are r×s (r is the height of the weights, s is the width of the weights), the convolution result can be represented as F(m×n, r×s), where m×n is the size of the output neuron image, m is the height of the output neuron image, and n is the width of the output neuron image. In order to reduce hardware complexity, improve versatility, and achieve good acceleration, embodiments of the present invention use convolution kernels (i.e., weights) of no more than 3×3 as base convolution units to perform Winograd convolution operations of arbitrary size with a stride of 1. Embodiments of the present invention decompose any F(m×n, r×s) into calculations of five types of base convolutions with operation sizes of 3×3, 3×2 (or 2×3), 3×1 (or 1×3), 2×2, and 2×1 (or 1×2), and then combine them. More specifically, embodiments of the present invention decompose any F(m×n, r×s) into a combination of basal convolution calculations of F(2×2, 3×3), F(2×2, 3×2), F(2×2, 2×3), F(2×2, 3×1), F(2×2, 2×2), and F(2×2, 2×1). It should be noted that since 1×1 convolution operations cannot be accelerated using Winograd convolution, the 1×1 size is not included in the basal convolution units defined in embodiments of the present invention.

[0038] Taking an input neuron image of size 6×6 and stride 1 as an example, F(2×2, 5×5), before performing Winograd convolution acceleration computation using the computing device of this embodiment, the 6×6 input neuron image and the 5×5 convolution kernel need to be linearly split based on the base convolution unit. The splitting process is as follows: Figure 1 As shown.

[0039] Figure 1The diagram illustrates a 5×5 convolution kernel 101 convolved with a 6×6 input neuron image 102 to obtain a 2×2 convolution result 103. The convolution kernel 101 needs to be split into sizes such as 3×3, 3×2 (or 2×3), 3×1, 2×2, and 2×1. In this embodiment, 3×3 is preferred, followed by 3×2 (or 2×3), then 3×1, then 2×2, and finally 2×1. According to this rule, convolution kernel 101 is split into four base convolution kernels: a 3×3 first base convolution kernel 104, a 3×2 second base convolution kernel 105, a 2×3 third base convolution kernel 106, and a 2×2 fourth base convolution kernel 107. That is, F(2×2, 5×5) is decomposed into one F(2×2, 3×3), one F(2×2, 3×2), one F(2×2, 2×3), and one F(2×2, 2×2). The input neuron image 102 is also correspondingly split into four sub-neuron data: a 4×4 first sub-neuron data 108, a 4×3 second sub-neuron data 109, a 3×4 third sub-neuron data 110, and a 3×3 fourth sub-neuron data 111.

[0040] Next, Winograd convolution operations are performed: the first base convolution kernel 104 is convolved with the first sub-neuron data 108 to produce the first sub-convolution result 112; the second base convolution kernel 105 is convolved with the second sub-neuron data 109 to produce the second sub-convolution result 113; the third base convolution kernel 106 is convolved with the third sub-neuron data 110 to produce the third sub-convolution result 114; and the fourth base convolution kernel 107 is convolved with the fourth sub-neuron data 111 to produce the fourth sub-convolution result 115.

[0041] Finally, the results of the first sub-convolution (112), the second sub-convolution (113), the third sub-convolution (114), and the fourth sub-convolution (115) are added together to obtain the convolution result 116, which is the same as the convolution result 103. This is an example of using the Winograd convolution algorithm to implement the original convolution operation.

[0042] Furthermore, the Winograd algorithm can be expressed by the following formula:

[0043] Y = A T [(GgG T )⊙(B T dB)]A

[0044] Where Y represents the output matrix of the convolution operation, and A T G is the inverse transformation multiplied by a constant matrix on the left, G is the weight transformation multiplied by a constant matrix on the left, and g is the weight of the original convolution. T The weight transformation is a right multiplication of a constant matrix, where ⊙ denotes positional multiplication, B TLet d be the left-multiplied constant matrix of the neuron transformation, d be the neuron data, B be the right-multiplied constant matrix of the neuron transformation, and A be the right-multiplied constant matrix of the inverse transformation. The left-multiplied and right-multiplied matrices of each transformation are simply transposes.

[0045] Taking F(2×2, 3×3) as an example, the aforementioned constant matrices are shown below:

[0046]

[0047]

[0048]

[0049] Figure 2 The diagram illustrates the transformation of the original convolution of F(2×2, 3×3) into a Winograd convolution. As shown, neuron data 201 is convolved with convolution kernel 202. During computation, neuron data 201 is arranged in a row according to the elements in sliding window 203. Sliding window 203 slides four times to form a 4×9 matrix 204. Then, the elements of convolution kernel 202 are arranged in a column to form a 9×1 matrix 205. The 4×9 matrix 204 and the 9×1 matrix 205 are convolved to obtain a 4×1 convolution result 206.

[0050] Next, based on the dotted lines in the diagram, the 4×9 matrix 204 is transformed into a 2×3 matrix 207, the 9×1 matrix 205 into a 3×1 matrix 208, and the 4×1 convolution result 206 into a 2×1 convolution result 209. After the linear transformation, the first element of the 2×1 convolution result 209 is R0 = M0 + M1 + M2, and R1 = M1 - M2 - M3. M0, M1, M2, and M3 can be represented by the following formula:

[0051] M0 = (K0 - K2)·W0

[0052] M3=(K1-K3)·W2

[0053] Through the aforementioned segmentation and linear transformation, the original convolution operation involved 36 multiplications, while the Winograd algorithm only requires 16 multiplications, reducing the computational complexity of multiplication by 2.25 times.

[0054] From the above transformation of the 2D convolution using the Winograd algorithm, it can be seen that the Winograd algorithm mainly consists of the following steps. First, the weights are multiplied by the weight constant matrix on the left and right, i.e., GgG TThis yields the weights after the Winograd linear transformation, i.e., the Winograd weight data. Next, a forward transformation operation is performed on the neuron data, which involves left and right multiplication of the neuron constant matrix, i.e., B... T dB, obtaining the forward transform data after the Winograd linear transform. Then, a positional multiplication operation is performed between the forward transform data and the Winograd weight data matrix, i.e., (GgG... T )⊙(B T dB) is used to obtain the positional multiplication data. Finally, the positional multiplication data is subjected to an inverse transformation operation, which is the left and right multiplication of the Winograd inverse transformation constant matrix, i.e., A T LA, where L is [(GgG T )⊙(B T This process, using dB), ultimately yields a convolution result equivalent to the original convolution.

[0055] From a hardware design perspective, the embodiments of this invention, taking into account the dependencies and operational differences among the three processes described above, streamline these three major transformation steps to achieve more efficient acceleration performance. The design of the forward transformation operation, the positional multiplication operation, and the inverse transformation operation will be described separately below.

[0056] Embodiments of the present invention utilize a forward transform unit to implement the forward transform operation, that is, to execute B. T dB, according to the rules of Winograd convolution, is a forward transform multiplied by matrix B on the left. T The size of the matrix is ​​(m+r-1)×(m+r-1), and the size of the right-multiplied matrix B is (n+s-1)×(n+s-1). Since the forward transformation is left-multiplied by matrix B... T Since the elements of the matrix B being multiplied by the right matrix consist of 0, 1, and -1, the matrix multiplication operation of the forward transformation can be decomposed into a fixed-pattern addition operation. The computing device in this embodiment of the invention is configured with a specific number of floating-point adders to complete the linear addition operations required for the entire matrix multiplication. Because this embodiment of the invention converts any original convolution into a basal convolution for calculation, the size of the forward transformation unit is related to the operation of the aforementioned five basal convolution sizes. Therefore, the following will use a 2×2 convolution result as an example (i.e., m = n = 2) to explain the data for the aforementioned five basal convolution operations FP32.

[0057] Taking a 3×3 base convolution as an example, its It can be represented as:

[0058]

[0059] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit directly corresponds to the number of adders, which is 4×(n+s-1)+4×(m+r-1)=32flops (floating-point operations per second). The input and output quantities of the forward transformation unit are (r+1)(s+1)×32=16×32 bits, respectively. The reason for multiplying the formula by 32 bits is for FP32 data, which is a 32-bit sequence. When the input and output quantities of the forward transformation unit are the same as the operation time, the hardware utilization of the forward transformation unit is most ideal. Therefore, the ratio of the input / output bandwidth of the forward transformation unit to the addition operation is preferably 16∶32=1∶2. In other words, when the cache bandwidth (or vectorization length) is l, the input bandwidth and output bandwidth of the forward transformation unit are l×32 bits, and the computing power of the adder group of the forward transformation unit is 2×l flops. Each operation produces 16 final results. Considering that 8 intermediate results will be generated during the operation, the minimum number of registers in the register file is 1×32×(16+8).

[0060] Taking a 3×2 base convolution as an example, its It can be represented as:

[0061]

[0062] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit is 4×(n+s-1)+2×(m+r-1)=20flops, while the input and output quantities of the forward transformation unit are (r+1)(s+1)×32=12×32 bits respectively. To improve the hardware utilization of the forward transformation unit, the optimal ratio of the input / output bandwidth to the addition operation is 12∶20=3∶5. That is, the input bandwidth and output bandwidth are 1×32 bits, and the computing power of the adder group is… Each calculation produces 12 final results and 6 intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(12+6).

[0063] Taking a 2×2 base convolution as an example, its It can be represented as:

[0064]

[0065] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit is 2×(n+s-1)+2×(m+r-1)=12flops. The input and output quantities of the forward transformation unit are (r+1)(s+1)×32=9×32 bits each. Therefore, the optimal ratio of the input / output bandwidth of the forward transformation unit to the addition operation is 9∶12=3∶4. That is, the input bandwidth and output bandwidth are 1×32 bits, while the computing power of the adder group is… Each calculation produces 9 final results and 6 intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(9+6).

[0066] Taking a 3×1 base convolution as an example, its It can be represented as:

[0067]

[0068] Based on the above formula, the forward transformation unit requires 4 flops of computing power, while the input and output quantities of the forward transformation unit are (r+1)×32=4×32 bits each. Therefore, the optimal ratio of the input / output bandwidth of the forward transformation unit to the addition operation is 4∶4=1∶1. That is, the input and output bandwidths are l×32 bits, and the computing power of the adder group is l flops. Each calculation produces 4 final results and 2 intermediate results. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is l×32×(4+2).

[0069] Taking a 2×1 base convolution as an example, its It can be represented as:

[0070]

[0071] Based on the above formula, the forward transformation unit's computational power requirement is 2 flops. The input and output quantities of the forward transformation unit are (r+1)×32 = 3×32 bits each. Therefore, the optimal ratio of the input / output bandwidth to the addition operation is 3:2. That is, the input and output bandwidths are 1×32 bits, while the computational power of the adder group is... Each calculation produces 3 final results and 1 intermediate result. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(3+1).

[0072] To simultaneously satisfy and support the aforementioned five types of base convolution operations, the embodiments of the present invention select that the input bandwidth and output bandwidth of the forward transform unit are the same, and the computing power of the addition operation is twice that of the input bandwidth and output bandwidth, that is, the input bandwidth and output bandwidth are both 1×32 bits, while the computing power of the adder group is 2×100 lops, and the number of register files is 1×32×(16+8).

[0073] Next, consider the positional multiplication and accumulation operator. Based on a comprehensive consideration of hardware design, scheduling strategy and execution performance, the embodiments of the present invention combine the positional multiplication operation and the feature map direction of the convolutional neuron data. By using the same positional multiplication and accumulation operator, it can not only effectively reduce the overall complexity of hardware design and resource consumption, but also reduce the number of on-chip cache accesses, save power consumption and area, and improve performance.

[0074] Assume the parameters of the convolutional layer are: input batch size N, input neuron channel size Ci, input neuron data height Hi, input neuron data width Wi, output neuron channel size Co, output neuron data height Ho, output neuron data width Wo, convolutional kernel size r×s, and stride 1. Since this embodiment supports F(2×2, r×s) operations, Ho = Hi - r + 1, Wo = Wi - s + 1, and the number of Winograd operation units... Where T is the number of blocks along the HW direction.

[0075] Due to the limited on-chip cache capacity, the computing device in this embodiment performs calculations in a single batch (N=1). Therefore, the scale of the input neuron data input to the computing device is [1Ci Hi Wi], the scale of the forward transformation data is [1Ci T(r+1)×(S+1)], the scale of the original weights is [Co Cirs], and the scale of the Winograd weight data is [1Co Ci(r+1)×(S+1)].

[0076] Figure 3 The diagram illustrates the aforementioned positional multiplication operation. Since N=1, the aforementioned data can be reduced to three dimensions. The size of the forward transformation data 301 is [Ci T(r+1)×(S+1)], with its three dimensions being Ci, T (i.e., the number of HW blocks) and (r+1)×(S+1), respectively. The Winograd weight data 302 is [Co Ci(r+1)×(S+1)], with its three dimensions being Co, Ci and (r+1)×(S+1), respectively. The positional multiplication operation involves cross-multiplying Co in the HW direction and accumulating in the Ci direction to obtain the positional multiplication data 303, with a size of [Co T(r+1)×(S+1)], with its three dimensions being Co, T and (r+1)×(S+1), respectively.

[0077] More specifically, the forward transformation data 301 involves T [Ci(r+1)×(S+1)] data units multiplied bitwise, and the Winograd weight data 302 involves Co [Ci(r+1)×(S+1)] data units multiplied bitwise to obtain an intermediate result of [Ci(r+1)×(S+1)]. Then, the result is accumulated along the Ci direction. This process is identical to matrix multiplication, thus it can be combined into a single matrix multiplication operation, making more efficient use of hardware resources and reducing the consumption of intermediate storage register resources.

[0078] Since the forward transform data 301 includes T data units of [Ci(r+1)×(S+1)], and the Winograd weight data 302 includes Co data units of [Ci(r+1)×(S+1)], each data unit of the forward transform data 301 needs to be multiplied by each data unit of the Winograd weight data 302. For example... Figure 4 As shown, during the positional multiplication operation, one data unit 401 of the forward transformation data 301 is homogeneously operated on with Co weight data units, i.e., the Co direction is used as the direction of parallel computation, and an intermediate result 402 is produced. Then, the next data unit is taken from the forward transformation data 301 and homogeneously operated on with Co weight data units again, producing the next intermediate result. This process is repeated until all T data units have been calculated, thus obtaining the positional multiplication data 303.

[0079] When the aforementioned data units are multiplied in pairs and accumulated along the feature map direction, the required computation is (Ci + Ci-1) × (r+1) × (S+1) floats. Since the value of Ci is often very large, it is difficult to use it as the granularity input to the pair multiplication and accumulation operator in practice. Therefore, this embodiment can further split Ci, perform multiplication and accumulation operations in units of vectorized length l, and split the multiplication and accumulation operation of the other dimension (r+1) × (S+1) into (r+1) × (S+1) steps to be completed sequentially. Finally, all results are added along the Ci direction to obtain the final result.

[0080] Since the output bandwidth of the forward transform unit is l×32 bits, to ensure that the overall pipeline time from the forward transform unit to the bitwise multiply-accumulate unit is the same, in this embodiment, the computing power of each bitwise multiply-accumulate unit in the bitwise multiply-accumulate unit is set to l+(l-1)flops, which includes l multiplication operations and l-1 addition operations. If the bitwise multiply-accumulate unit has ω parallel dimensions, that is, including ω bitwise multiply-accumulate units operating simultaneously, then the computing power of the bitwise multiply-accumulate unit is ω×(l+(l-1))flops, which is a function of ω and l.

[0081] This embodiment also includes an inverse transformation unit to perform the inverse transformation operation, which involves left-multiplying the matrix according to the inverse transformation. and right multiplication of matrix A (n+s-1)×2 Perform A T LA calculation, where L is (GgG T )⊙(B T dB). Due to the inverse transformation, the matrix A is multiplied on the left. T Since the elements of the right-multiplied matrix A are also composed of 0, 1, and -1, the matrix multiplication operation of the inverse transform can also be decomposed into a fixed-pattern addition operation. The adder group of the inverse transform unit is configured with a specific number of floating-point adders to complete the linear addition operations required for the entire matrix multiplication. The following explanation also considers the five types of base convolution to determine the size of the inverse transform unit.

[0082] Taking a 3×3 base convolution as an example, its It can be represented as:

[0083]

[0084] Based on the above formula, the inverse transform computing power of ITU 915 is 24f10ps, while the input bandwidth is (r+1)(s+1)×32=16×32 bits, and the output bandwidth is (s+1)×32=4×32 bits. Similarly, when the input bandwidth and computing power of the inverse transform unit are the same, the hardware utilization of the inverse transform unit is optimal. Therefore, the optimal ratio of input bandwidth to addition operation is 16∶24=2∶3, that is, the input bandwidth is 1×32 bits, and the computing power of the adder group is… Each calculation produces 16 final results, with no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×16.

[0085] Taking a 3×2 base convolution as an example, its It can be represented as:

[0086]

[0087] Based on the above formula, the inverse transform unit has an inverse transform computing power of 16 flops, an input bandwidth of 12 × 32 bits, and an output bandwidth of 4 × 32 bits. The optimal ratio of input bandwidth to addition operation is 12:16 = 3:4, meaning the input bandwidth is 1 × 32 bits, while the computing power of the adder group is... Flops produces 12 final results per calculation and no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×12.

[0088] Taking a 2×2 base convolution as an example, its It can be represented as:

[0089]

[0090] Based on the above formula, the inverse transform unit has an inverse transform computing power of 10 flops, an input bandwidth of 9 × 32 bits, and an output bandwidth of 4 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 9:10, i.e., the input bandwidth is 1 × 32 bits, while the computing power of the adder group is... Each calculation produces 9 final results, with no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×9.

[0091] Taking a 3×1 base convolution as an example, its It can be represented as:

[0092]

[0093] Based on the above formula, the inverse transformation unit has an inverse transformation computing power of 4 flops, an input bandwidth of 4 × 32 bits, and an output bandwidth of 2 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 4:4 = 1:1, that is, the input bandwidth is 1 × 32 bits, and the computing power of the adder group is 1 flops. Each calculation will produce 4 final results and 2 intermediate results. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is 1 × 32 × (4 + 2).

[0094] Taking a 2×1 base convolution as an example, its It can be represented as:

[0095]

[0096] Based on the above formula, the inverse transform unit has an inverse transform computing power of 2 flops, while the input bandwidth is 3 × 32 bits and the output bandwidth is 3 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 3:2, i.e., the input bandwidth is 1 × 32 bits, and the computing power of the adder group is... Each calculation produces 3 final results and 1 intermediate result. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(3+1).

[0097] To simultaneously satisfy and support the aforementioned five types of base convolution operations, the computational power of the addition operation in the inverse transform unit can be set to a value equal to the input bandwidth. This means that when the input bandwidth is 1×32 bits, the computing power of the adder group is [number] times ...

[0098] However, to simplify the hardware design, this embodiment can further consider having the same hardware configuration for the forward and inverse transform units. While simultaneously meeting the requirements of both the forward and inverse transform units, this embodiment chooses to use the forward transform unit design for the inverse transform unit, meaning the input and output bandwidths are the same, and the computational power of the addition operation is twice that of the input and output bandwidths. In other words, the inverse transform unit has an input bandwidth of 1×32 bits and an output bandwidth of 1×32 bits, resulting in a computational power of 2×1 flops for the adder group.

[0099] In summary, the bandwidth and computing power of the three core modules (forward transform unit, bitwise multiplication accumulator, and inverse transform unit) performing Winograd convolution operations in this embodiment have been matched. That is, the input bandwidth of each of the three core modules is set to 1×32 bits, and the output bandwidth is also set to 1×32 bits. The computing power of the forward transform unit is 2×1 flops, the computing power of the bitwise multiplication accumulator is ω×(1+(1-1)) flops, and the computing power of the inverse transform unit is 2×1 flops.

[0100] As mentioned above, the Winograd convolution operation is directly related to the vectorization length parameter *l*. The vectorization length parameter *l* is the minimum processing length and relates to the reuse of neuron transformations in this embodiment's computing device. A larger *l* parameter results in a higher reuse rate, while the required memory access, computational load, power consumption, and average hardware design area decrease proportionally. However, the parameters of the neural network convolutional layers change with the network model. As the vectorization length parameter *l* increases, when some network models have fewer channels than *l*, computational power is wasted, affecting the acceleration effect and leading to additional overhead in area and power consumption. Therefore, a trade-off analysis of these two factors is necessary when determining the vectorization length *l* to plan the most suitable vectorization length parameter configuration.

[0101] Based on empirical values, weights were assigned to several key hardware components in this embodiment (such as the FP32 adder, bitwise multiplication unit, registers, etc.) to obtain their computational power and resource overhead functions. It was found that when l is greater than 16, the utilization rate of hardware resources can be guaranteed to be at a high level. Furthermore, the number of input and output channels of commonly used neural network models (such as LeNet, VGG16, VGG19, AlexNet) were considered, and their computational power loss was calculated. It was found that when l is greater than 64, the overall computational power loss will increase significantly. From the above two quantitative analyses, it can be concluded that when the vectorization length parameter l is between 16 and 64, the computing device of this embodiment performs well. If versatility is further considered to meet possible future network structures and parameters, this embodiment preferably chooses l = 16.

[0102] Figure 5A schematic diagram of the structure of the aforementioned embodiment, presented in the form of a circuit board, is shown. Figure 5 As shown, board 50 includes chip 501, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 50 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0103] Chip 501 is connected to external device 503 via external interface device 502. External device 503 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 503 to chip 501 via external interface device 502. The calculation results from chip 501 can be transmitted back to external device 503 via external interface device 502. Depending on the application scenario, external interface device 502 may have different interface forms, such as a PCIe interface.

[0104] The board 50 also includes a storage device 504 for storing data, which includes one or more memory cells 505. The storage device 504 is connected to and transmits data with the controller 506 and the chip 501 via a bus. The controller 506 in the board 50 is configured to regulate the state of the chip 501. Therefore, in one application scenario, the controller 506 may include a microcontroller (MCU).

[0105] Figure 6 This is a structural diagram illustrating the combined processing device in chip 501 of this embodiment. (As shown...) Figure 6 As shown, the combined processing device 60 includes a computing device 601, an interface device 602, a processing device 603, and a DRAM 604.

[0106] The computing device 601 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor, to perform deep learning or machine learning calculations, especially Winograd convolution operations. It can interact with the processing device 603 through the interface device 602 to jointly complete the user-specified operations.

[0107] Interface device 602 is used to transmit data and control commands between computing device 601 and processing device 603. For example, computing device 601 can obtain input data from processing device 603 via interface device 602 and write it into on-chip cache of computing device 601. Further, computing device 601 can obtain control commands from processing device 603 via interface device 602 and also write them into on-chip cache of computing device 601. Alternatively or optionally, interface device 602 can also read data from on-chip cache of computing device 601 and transmit it to processing device 603.

[0108] The processing device 603, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 601. Depending on the implementation, the processing device 603 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 601 of this invention can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 601 and the processing device 603 are considered together, they are considered to form a heterogeneous multi-core structure.

[0109] DRAM 604 is used to store data to be processed. It is off-chip memory and is typically 16GB or larger in size. It is used to store the data required by computing device 601 and / or processing device 603, especially the neuron data and weights to be used for Winograd convolution operations.

[0110] Before performing Winograd convolution in this embodiment, data preprocessing and instruction preprocessing are required for the Winograd algorithm to ensure that the Winograd algorithm can run correctly and efficiently on the computing device 601. The relevant preprocessing work is described below.

[0111] First, padding needs to be considered. Unlike the padding operation of the original convolution, the padding of the Winograd convolution must ensure that the padded neuron data can be completely segmented into data units and satisfy the input-output size parameters. For convolutions with an odd input size of neuron data, the processing device 603 needs to pad an extra row and a column of elements to make the neuron input size even. Correspondingly, the convolution result produced by the Winograd convolution will also have more elements.

[0112] Figure 7 This diagram illustrates the preprocessing of neuron data with two Vickers elements, including padding. The original convolutional neuron data 701 comprises two parts: a neuron image 702 and padding 703. The neuron image 702 is the original 5×5 neuron data, and padding 703 is performed before the original convolution, transforming it into neuron data 701. Neuron data 701 is convolved with weights to generate the original convolution result 704. As shown, the convolution result 704 is a 5×5 data matrix. In this embodiment, when the original convolutional neuron data 701 is to be converted into neuron data suitable for Winograd convolution, since the number of elements in neuron data 701 is odd (7×7), the processing unit 603 adds a row and a column of padding 705 outside of padding 703, making it neuron data 706 with an even number of elements (8×8). The computing device 601 performs Winograd convolution based on neuron data 706 and Winograd weight data to generate the convolution result 707. As can be seen from the figure, the convolution result 707 has an extra row and a column of extra elements 708 compared to the convolution result 704. The extra elements 708 will eventually be discarded, making the results of the two convolution methods consistent.

[0113] As for dimensions that already have an even number of elements, the processing device 603 does not add extra padding; it simply maintains the original input size as a non-odd dimension. In dimensions with an even number of elements, the original convolution result is the same size as the Winograd convolution result.

[0114] Next, consider the base convolution instruction scheduling scheme. As mentioned earlier, this embodiment will split all convolution kernels into five types of base convolution units: 3×3, 3×2 (or 2×3), 2×2, 3×1 (or 1×3), and 2×1 (or 1×2). To accelerate computation, this embodiment uses as few consistent base convolutions as possible during instruction scheduling. Therefore, during preprocessing, the processing device 603 needs to split the kernels into as many identical base convolution units as possible based on their size, and then select the splitting results in the following priority order: 3×3 is better than 3×2 (or 2×3), then better than 2×2, then better than 3×1 (or 1×3), and finally 2×1 (or 1×2). For example, a 6×6 convolution kernel can be split into 4 3×3 basal convolution units, or 9 2×2 basal convolution units, or 2 3×3 basal convolution units plus 3 2×3 basal convolution units, etc. In this case, the processing device 603 preferentially selects 4 3×3 basal convolution units, which consumes the least resources.

[0115] Finally, consider the transformation of the original weights. In this embodiment, the computing device 601 only accepts weight data after Winograd transformation; therefore, the processing device 603 needs to pre-perform the Winograd transformation on the original weights. Since the computing device 601 supports only five types of basis convolution, when the processing device 603 transforms the original weight data into Winograd weight data, it needs to design complete implementation programs for all five transformations. For one-dimensional basis convolution, the weight transformation can be completed by directly performing the addition required for the corresponding matrix operations; for two-dimensional or more-dimensional basis convolution, the weight transformation is the same as the forward / inverse transformation of the neuron. The processing device 603 first performs a left multiplication matrix operation for the weight transformation, and then performs a right multiplication operation, i.e., executes GgG. T GgG T The calculated result is the Winograd weight data.

[0116] Finally, consider the reconstruction of the input data storage dimension. As mentioned earlier, for the computing device 601, the data arrangement has the following requirements. Since neuron data is transmitted in C... i The dimensional direction is vectorized, and data is processed in 1×32-bit units. This requires that neuron data be stored in DRAM 604 along the C-axis. i The data is stored contiguously in the direction of the DRAM 604, preferably aligned to a 64-byte data unit. Based on these requirements, the processing device 603, referring to the structure of the DRAM 604, reconstructs the data storage of the neuron data according to the aforementioned order and vector width. During Winograd convolution calculations, the Winograd weight data is also sent from the DRAM 604 to the computing device 601 in C... iSince the dimension direction is used as the vectorization direction and is performed in l×32-bit data units, the order in which the processing device 603 stores the Winograd weight data into the DRAM 604 also needs to directly correspond to the on-chip operation order. This is to ensure that the input / output and on-chip operation pipelined execution achieve the highest performance.

[0117] In summary, this embodiment provides a method for preprocessing neuron data and weight data of Winograd convolutions. Figure 8 The flowchart is shown.

[0118] First, the neuron data is padded. In step 801, the processing device 603 determines whether the input size of any dimension in the neuron data is odd. The input size refers to the number of elements padded after performing the original convolution, i.e., neuron data 701. If it is odd, step 802 is executed, where the processing device 603 padded the dimensions with odd input sizes to make the input size even. For example, in two dimensions, a row and a column are padded based on neuron data 701 to form neuron data 706. If it is even, step 803 is executed, maintaining the dimensions with non-odd input sizes.

[0119] Next, the basal convolution instruction scheduling is performed. In step 804, the processing device 603 sets the size of the basal convolution, that is, it splits and selects appropriate combinations of basal convolutions according to the kernel size. In step 805, the processing device 603 converts the original weight data into Winograd weight data according to the selected basal convolution size.

[0120] Finally, the storage dimension of the input data is reconstructed. In step 806, the processing device 603 sets the size of the data unit to the product of the vectorization direction and the number of data bits, i.e., 1×32 bits. In step 807, the processing device 603 aligns the neuron data in units of data units. In step 808, the processing device 603 aligns the Winograd weight data in units of data units. In step 809, the processing device 603 stores the preprocessed neuron data and Winograd weight data in DRAM 604, and DRAM 604 stores the neuron data and Winograd weight data in units of data units. At this point, the preprocessing work for the Winograd convolution operation is complete.

[0121] Figure 9A structural diagram of computing device 601 is shown. Computing device 601 includes a bus 901, a direct memory access (DMA) module 902, an instruction cache (IRAM) 907, a decoding unit (IDU) 908, a neuron cache (NRAM) 909, a forward transformation unit (NTU) 910, a forward transformation data cache (WNRAM) 911, a weight cache (WRAM) 912, a positional multiplication accumulator (MAC) unit 913, a positional multiplication data cache (WRRAM) 914, an inverse transformation unit (ITU) 915, a result cache (RRAM) 916, and an arithmetic logic unit (ALU) 917.

[0122] Bus 901 is a common communication trunk line for transmitting information between devices. It is a transmission line bundle composed of wires. According to the types of information transmitted by the combined processing device 60, bus 901 is a collective term for data bus, address bus, and control bus, used to transmit data, data address, and instructions respectively. Bus 901 serves as the communication channel between DRAM 604 and computing device 601, and in this embodiment, it is specifically PCIe.

[0123] The DMA module 902 is used to copy data from one address space to another, typically moving data between external memory (such as DRAM 604) and the internal cache of the computing device 601. During DMA transfer, the processing device 603 relinquishes bus control to the DMA module 902, which controls the bus 901 to perform data transfer. After the DMA transfer is complete, the DMA module 902 returns bus control to the processing device 603.

[0124] DMA module 902 includes Neuron Direct Memory Access (NDMA) 903, Weight Direct Memory Access (WDMA) 904, Instruction Direct Memory Access (IDMA) 905, and Result Direct Memory Access (RDMA) 906. NDMA 903 is used to input neuron data from DRAM 604, WDMA 904 is used to input Winograd weight data from DRAM 604, IDMA 905 is used to input instructions from DRAM 604, and RDMA 906 is used to output calculation results to DRAM 604. In other embodiments, NDMA 903, WDMA 904, IDMA 905, and RDMA 906 can be implemented using the same direct memory access module.

[0125] IRAM 907 is used to temporarily store instructions input from IDMA905. IDU908 fetches instructions from IRAM 907, decodes them, and controls the operation of other units based on the decoded instructions. IDU 908 is the decoding and scheduling unit of the entire computing device 601. It is responsible for decoding the control instructions obtained from DRAM 604, converting them into control signals to coordinate the operation of various modules / units on the chip, and is also responsible for instruction order preservation, dependency removal, branch prediction, exception handling, interrupt handling, and many other tasks. Figure 9 Thin arrows indicate control flow, while thick arrows indicate data flow.

[0126] Since the computing device 601 is primarily designed for Winograd convolution calculations and lacks general-purpose processing capabilities or has limited general-purpose processing capabilities, it relies heavily on the scheduling and data communication of the processing device 603 during task execution. This results in frequent input / output communication between the computing device 601 and the processing device 603, significantly limiting the performance of the computing device 601. To address this, the computing device 601 incorporates multiple small-capacity on-chip caches to buffer reusable temporary data, such as NRAM 909, WNRAM 911, WRAM 912, and WRRAM 914.

[0127] During on-chip and off-chip data transfer, neuron data and Winograd weight data are transferred in single batches (N=1). Specifically, the data unit for neuron data is [Ci Hi Wi], while the data unit for Winograd weight data is [Co Ci(r+1)×(s+1)]. The result after the Winograd convolution operation is [CoHo Wo]. The former two are input data, and the latter is output data. These represent the minimum throughput for transmission and computation in computing device 601. The actual data throughput depends on the size of the on-chip cache and the computation scheduling process, which will be further explained below.

[0128] Based on the characteristics of convolution operations, it is known that the convolution operations involved in input data of the above scale can be split in multiple dimensions, such as in the Ci direction, the HW image direction, or the Co direction. However, when Winograd transformation is involved, the smallest operational splitting unit is F(2×2, r×s), and the smallest splitting unit in the HW direction is (r+1)×(s+1). Considering that the base convolution size for Winograd acceleration implemented by the computing device 601 does not exceed 3×3, this embodiment estimates the cache capacity based on the 3×3 base convolution, which consumes the most on-chip cache resources.

[0129] According to the rules of Winograd convolution, during the forward transformation operation, parallel processing is required in the Ci direction in units of vectorized length parameter l. During the positional multiplication and accumulation operation, parallel processing is required in the Co direction in units of l. During the inverse transformation, parallel processing is required in the Co direction in units of l. Therefore, the minimum input data block size of the neuron participating in the operation can be deduced to be [l(r+1)×(s+1)]. Since it is estimated using 3×3 radix convolution, the size of the neuron transformation result data block is [l4×4]. The size of the Winograd weight data block that needs to be multiplied and accumulated with it is [l l4×4], the size of the positional multiplication output data block is [l4×4], and the size of the inverse transformation output result is [l2×2].

[0130] While designing an on-chip cache at this scale can meet all requirements and takes into account the design principles of reuse and low power consumption, the above-mentioned data only represents the minimum input / output storage data size for implementing the functionality. Further optimization potential for the input / output volume of Winograd convolution operations needs to be considered. This embodiment further plans the cache as follows.

[0131] During the positive transformation of the neuron, the operation is based on F(2×2, r×s) and l as the smallest implementation unit of vectorization length. The size of the data block retrieved each time is [l 4 4], and the step size of the neuron in retrieving data is kept at 2. For example Figure 10 As shown, there is a quarter-overlapping portion 1006 between the data unit 1001 to be transformed and the four data blocks 1002, 1003, 1004, and 1005 generated by the sliding window. The size of the overlapping portion 1006 is [l 4 4]. As can be seen from the figure, during the forward transformation of the data unit 1001, each of the data blocks 1002, 1003, 1004, and 1005 includes one overlapping portion 1006, thus generating a total of four overlapping portions 1006. When the data is moved by splitting it according to the smallest data unit [l 4 4], the data throughput required for the overlapping portion 1006 increases fourfold, resulting in increased redundant data. To solve this problem, this embodiment further reduces the input / output by caching larger data units in the on-chip cache of the computing device 601.

[0132] As mentioned earlier, neuron data of size [Ci Hi Wi] is convolved with Winograd weight data of size [Co Ci(r+1)×(s+1)]. This embodiment retains as much Winograd weight data as possible on-chip, i.e., temporarily stores as much as possible on-chip. With [ll(r+1)×(s+1)], a batch of neuron data can be calculated with only one weight loading operation, thus saving the amount of weight data input / output.

[0133] For the output data, since the convolutional neural network also involves other network layer operations such as activation, pooling, and normalization, the convolution results need to be cached on-chip for subsequent network layer operations. Therefore, the computing device 601 reserves a fixed-capacity cache to store the convolution results. This cached data can share the cache space with the final results after various other layer operations, thus reducing the data throughput of reloading the convolution results for other layer operations and transmitting the computation results.

[0134] The above optimization analysis shows that the cache capacity for neuron data should be as large as possible to reduce the total throughput of neuron data. Furthermore, since neuron data is accumulated along the Ci direction, the larger the amount of data stored along the Ci direction, the fewer times reloading and re-accumulation are required. Additionally, the cache space for Winograd weight data also needs to be as large as possible. Finally, this embodiment also needs to reserve corresponding output result space for other layer operations. In summary, this embodiment divides the on-chip cache into three main parts, each responsible for different functions: Nram 909 stores neuron data, Wram 912 stores Winograd weight data, and Rram 916 stores convolution results. The computing device 601 further sets up two caches to temporarily store intermediate results: WNram 911 temporarily stores the data after forward transformation, and WRram 914 temporarily stores the data after bitwise multiplication and accumulation.

[0135] While a larger cache capacity is generally better for storing neuron data, Winograd weights, and convolution results, the cache size is closely related to the configuration of the computing unit resources. Excessive cache size will reduce the computing power of the computing device 601. The criterion for judgment is a balance between input / output bottleneck pressure and computational power pressure. In this embodiment, the NRAM 909 size is set to... Where α is The direction factor is α, where β is the direction factor of HW; the size of Wram 912 is set to α×γ×[11 4 4], where γ is the direction factor of HW. The direction coefficients; the size of Rram 916 is set to β×γ×[l 2 2]. The time required to complete the calculation of these sizes of data is l×α×β×γ.

[0136] Preferably, in this embodiment, l is chosen to be 16, α to be 4, β to be 64, and γ to be 16. Considering that the data size of each FP32 is 4B, the storage capacity of the NRAM 909 storage array is... The storage capacity of Wram 912 is α×γ×[l 4 4]×4B=1MB, and the storage capacity of Rram 916 is β×γ×[l 2 2]×4B=256KB.

[0137] Back Figure 9 The NRAM 909, based on the decoded instructions, temporarily stores the neuron data sent by the NDMA 903. The NTU910, based on the decoded instructions, reads the neuron data from the NRAM 909 and performs a forward transform, i.e., a B-mode transformation. T dB is calculated to generate positive transform data, which is then temporarily stored in WNram 911. Figure 11 A schematic diagram of the NRAM 909 is shown. In this embodiment, the NRAM 909 includes four memory arrays 1101, 1102, 1103, and 1104. Each memory array includes four memory blocks 1105, 1106, 1107, and 1108. Each memory block has a size of d w-bit storage cells, where d represents the number of addresses in the storage cell. Preferably, w is 128 and d is 1024, then the size of each storage block is 16KB, the size of each memory array is 64KB, and the total storage capacity of the NRAM 909 is 256KB. Its total width is 4 × w = 64B, and its depth is 4 × d = 4 × 1024.

[0138] In the width direction, the input bandwidth of the NRAM 909 is set to 4 bits, while the output bandwidth must match the input bandwidth of the NTU 910. As mentioned earlier, the input bandwidth of the NTU 910 is set to 1 × 32 bits, and 1 is preferably 16, so the input bandwidth of the NTU 910 is 64 bits. Therefore, the output bandwidth of the NRAM 909 is also 4 × w = 64 bits. The input and output of the NRAM 909 need to be performed simultaneously, so a dual-port design is adopted.

[0139] Figure 12 A schematic diagram of the NTU 910 is shown. The NTU 910 includes an input buffer 1201, a register file 1202, an adder set 1203, and an output buffer 1204.

[0140] When the NTU 910 receives an instruction to load neuron data from the NRAM 909, the input buffer 1201 acts as a first-in-first-out queue buffer to temporarily store neuron data based on the input bandwidth of 64B. The neuron data loading phase continues until all data reception is complete, and the entire process is controlled by instructions sent by the IDU 908.

[0141] According to the decoded instructions, register file 1202 retrieves the temporarily stored neuron data from input buffer 1201 and stores it in a specific address in register file 1202 in the planned order of operations. This neuron data stored in a specific address in register file 1202 becomes the addition operand. In this embodiment, since the pipeline time lengths of the input, operation, and output stages of the NTU 910 should be equal, a cache hardware resource dependency phenomenon will occur. To solve the resource dependency problem, the register file 1202 is divided into two storage units of the same size: ping storage unit 1205 and pong storage unit 1206. The i-th addition operand and the resulting positive transformation data are temporarily stored in the ping storage unit 1205, the (i+1)-th addition operand and the (i+1)-th positive transformation data are temporarily stored in the pong storage unit 1206, and the (i+5)-th addition operand and the (i+5)-th positive transformation data are temporarily stored in the ping storage unit 1205, overwriting the i-th addition operand and the (i+5)-th positive transformation data. The register file 1202 stores data according to this rule.

[0142] Adder group 1203 reads addition operands sequentially from specific addresses in register file 1202 according to the decoded instructions and performs addition operations. In this embodiment, there are two groups of adder groups 1203 to correspond to the addition operation scheduling direction. Each group includes 16 adders to correspond to the vectorization direction l. Each adder is an FP32 adder. It performs addition operations in the forward transform of the Winograd convolution in a specific order along the neuron data channel direction. This specific order is to first calculate the addition of the left multiplication matrix BT of the Winograd convolution, then calculate the addition of the right multiplication matrix B of the Winograd convolution, and finally generate the forward transform data, which is then stored back in register file 1202. The operation order, register allocation, and operation time are all related to the convolution filter size and are controlled by instructions sent by IDU 908. This operation stage is data-dependent with the aforementioned neuron data loading stage and is executed in a pipelined manner, implemented by hardware counting.

[0143] Output buffer 1204 is also a first-in-first-out queue buffer, used to temporarily store forward transformation data sequentially from ping storage unit 1205 and ping storage unit 1206. This output stage depends on the overall completion of the computation stage in order to output the corresponding buffered data based on the output bandwidth 64B.

[0144] To save overhead, the forward transformation data needs to be multiplexed, so the WNram 911 is configured for temporary storage and repeated transmission. It includes multiple buffer units. Figure 13A schematic diagram of an exemplary WNram 911 is shown. As shown, the WNram 911 includes four cache units: a first cache unit 1301, a second cache unit 1302, a third cache unit 1303, and a fourth cache unit 1304. Forward transformation data from the NTU 910 is sent to one or more of these cache units via routing.

[0145] The WNram 911 sends forward transform data to the MAC 913 in a specific order for subsequent calculations. The WNram 911 is designed to buffer a portion of the forward transform data, sending this data to the MAC 913 before storing the next portion of forward transform data, thus reducing the size of the WNram 911 through pipelining. Furthermore, considering that the forward transform data will be multiplied bitwise with Winograd weight data of size γ × [l l4×4], the WNram 911 is configured to package and send data to the MAC 913 in units of γ data blocks for calculation. In this way, forward transform data is only output on average every γ cycles, effectively reducing the power consumption of the WNram 911. Accordingly, the first γ forward transform data are sequentially overwritten by the next γ forward transform data, thus limiting the minimum storage size of the WNram 911 to [l(r+1)(s+1)] × 4B, which is [l 44] × 4B = 1KB as mentioned earlier.

[0146] Specifically, the width of the first cache unit 1301, the second cache unit 1302, the third cache unit 1303, and the fourth cache unit 1304 is w1 bytes, and the depth is d1, and they are divided into m parts in the depth direction. In this embodiment, m is preferably 8, w1 is 64, and d1 is 128. Therefore, the width of each cache unit is 64 bytes, the depth is 128, and the address space is divided into 8 parts in the depth direction for data reuse. The size of each cache unit is 8KB, that is, the total capacity of WNram 911 is set to 32KB.

[0147] Back Figure 9 Wram 912 temporarily stores the Winograd weight data sent by WDMA 904 according to the decoded instructions. MAC 913 reads the Winograd weight data from Wram 912 and the forward transform data from WNram 911 according to the decoded instructions, and performs bitwise multiplication and accumulation operations on the forward transform data and the Winograd weight data, that is, performs [(GgG T )⊙(B T The operation of dB)] is performed to generate the bitwise multiplication data, and the bitwise multiplication data is temporarily stored in WRram 914.

[0148] Figure 14A schematic diagram of Wram 912 is shown. In this embodiment, Wram 912 includes four storage arrays 1401, 1402, 1403, and 1404. WDMA 904 sends Winograd weight data to storage arrays 1401, 1402, 1403, and 1404 via routing. Each storage array includes four storage blocks 1405, 1406, 1407, and 1408, and each storage block includes four storage cells 1409, 1410, 1411, and 1412. The size of each storage cell is 4×d×w. As mentioned earlier, w is 128 and d is 1024, so the size of each storage block is 64KB, and the size of each storage array is 256KB. The total capacity of Wram 912 is 1MB. For each storage block, its width is 4×w=512 bits. It is divided into 4 independent address storage spaces in the depth direction, each segment with a depth of d=1024, and the total depth is 4×d=4096.

[0149] In this embodiment, each memory array 1401, 1402, 1403, and 1404 independently has an input bandwidth and output bandwidth of 4×wB, and the total output bandwidth of Wram 912 is 4×4×wB. Specifically, when w is 128, the input bandwidth and output bandwidth of each memory array are 64B, and the total output bandwidth and total output bandwidth are 256B.

[0150] In this embodiment, the MAC 913 includes 64 MAC processors, divided into 4 groups to perform 4 different batch operations, with each group of 16 MAC processors being independently distributed. The forward transform data of the WNram911 needs to be sent to these 64 MAC processors simultaneously, so that they can perform bitwise multiplication and accumulation operations with different Winograd weight data. Therefore, the WNram 911 sends the forward transform data by broadcasting or distributing routes. Due to the large output load, in order to ensure driving capability and timing, the forward transform data of the WNram911 is broadcast or distributed through two levels of N1 and N2 routes. First, it is sent to 4 N1 nodes, each N1 node broadcasts or distributes routes to 4 N2 nodes, and each N2 node then broadcasts or distributes routes to 4 MAC processors.

[0151] Figure 15 The diagram shows the output side of the WNram 911. The MAC 913 first performs bitwise multiplication, and then accumulates the resulting vectors sequentially. Its logical function is equivalent to calculating the dot product of vectors or performing element-wise operations in matrix multiplication. Each MAC group includes 16 MAC units 1501, i.e., ω = 16. Since l is preferably 16, the computing power of each MAC group is 16 × (16 + (16 - 1)) = 496 flops.

[0152] Figure 16 The diagram shows the output side of the Wram 912. The four outputs of the Wram 912 are responsible for data transmission to 16 MAC units 1501. In fact, in this embodiment, each storage array 1401, 1402, 1403, and 1404 is responsible for the data transmission to the 16 MAC units 1501 on a single N1 node. Since the output bandwidth is only 64 bytes, time-division multiplexing of the bandwidth is required. Each N2 node occupies only one-eighth of the bandwidth time, leaving the other half idle to reduce power consumption. More specifically, the Wram 912 sends the Winograd weight data to the N1 node using a 64-byte bandwidth. The N1 node then broadcasts the data to the N2 node using a 64-byte bandwidth. The N2 node then distributes the data to each MAC unit 1501 using a routing method, also using a 64-byte bandwidth. Each MAC unit 1501 can perform an FP32 multiply-accumulate operation of length l.

[0153] ITU 915 reads the bitwise multiplication data from WRram 914 according to the decoded instructions, performs the inverse transformation of the bitwise multiplication data, that is, performs A... T The LA operation is performed to obtain the convolution result, which is temporarily stored in Rram 916.

[0154] Figure 17 A schematic diagram of ITU 915 is shown. ITU 915 includes an input buffer 1701, a register file 1702, an adder group 1703, and an output buffer 1704.

[0155] When ITU 915 receives an instruction to load bitwise multiplication data from WRram 914, input buffer 1701 acts as a first-in-first-out queue buffer to temporarily store bitwise multiplication data based on the input bandwidth. The bitwise multiplication data loading phase continues until all data reception is complete. Convolutional filters of different sizes are configured with fixed and independent buffer resource allocations and input counts. The entire process is controlled by instructions sent by IDU 908.

[0156] According to the decoded instructions, register file 1702 retrieves the temporarily stored bitwise multiplication data from input buffer 1701 in a fixed order of operations and stores it at a specific address in register file 1702. These bitwise multiplication data stored at a specific address in register file 1702 become the addition operands. Similarly, to solve the resource dependency problem, register file 1702 has two storage units of the same size, ping-pong and ping-pong, ping-pong and ping-pong respectively. The i-th addition operand and the resulting convolution are temporarily stored in ping-pong storage unit 1705, the (i+1)-th addition operand and the (i+1)-th convolution result are temporarily stored in ping-pong storage unit 1706, and the (i+5)-th addition operand and the (i+5)-th convolution result are temporarily stored in ping-pong storage unit 1705, overwriting the i-th addition operand and the i-th convolution result. Register file 1702 stores data according to this rule.

[0157] Adder group 1703 reads addition operands sequentially from specific addresses in register file 1702 according to the decoded instructions and performs addition operations. Similar to adder group 1203, adder group 1703 consists of two groups to correspond to the addition operation scheduling direction. Each group includes 16 adders to correspond to the vectorization direction. Each adder is an FP32 adder, performing addition operations in the inverse transform of the Winograd convolution in a specific order along the channel direction of the bitwise multiplication data. This specific order is to first calculate the left multiplication matrix A of the Winograd convolution. T The addition is performed, followed by the addition of the right multiplication matrix A of the Winograd convolution, and finally the convolution result is generated and stored back into register file 1702. The order of operations, register allocation, and operation time are all related to the size of the convolution filter and are controlled by instructions sent by IDU 908. This operation stage is data-dependent with the aforementioned stage of loading bitwise multiplication data, and is executed in a pipelined manner, implemented by hardware counting.

[0158] Output buffer 1704 is also a first-in-first-out queue buffer, used to temporarily store the convolution results sequentially from ping-pong storage unit 1705 and ping-pong storage unit 1706. This output stage depends on the overall completion of the computation stage in order to output the corresponding buffered results based on the output bandwidth.

[0159] Besides Winograd convolution, computing device 601 can also perform all neural network-related operations. ALU 917 performs two main tasks based on the decoded instructions: the first is convolution fusion operations, which can be completed on-chip with the convolutional layers without relying on additional data. These operations include activation, biasing, orientation, and accumulation. The second task is non-convolutional operations. The results of the ALU 917 are also temporarily stored in RRAM 916. The presence of ALU 917 ensures that all operations in convolutional neural networks can be fully implemented in computing device 601, giving computing device 601 the versatility and completeness of neural networks.

[0160] According to the decoded instructions, the RDMA 906 retrieves the convolution result from the RRAM 916 and outputs it to the DRAM 604, thus completing the entire convolution operation. Similarly, the RDMA 906 can also retrieve other operation results generated by the ALU 917 from the RRAM 916 and output them to the DRAM 604 according to the decoded instructions. In this embodiment, the output bandwidth of the RRAM 916 is w bytes, and it also includes four memory arrays. Each memory array includes 4×d ​​4×w bit memory cells, i.e., a width of 512 bits and a depth of 4096. Therefore, the size of each memory array is 256KB, and the size of the RRAM 916 is 1MB. The input and output dual-port bandwidth of each memory array is 64B. The address space in the depth direction is divided into 16 parts, each with an address space of 256, used to store the results of the neuron multiplexing direction.

[0161] Figure 18 The diagram illustrates the connection relationships of the RAM 916. The input ports of the RAM 916 are connected to ITU 915 and ALU 917 to receive their output data. Since convolution and other operations are not performed, these two input ports do not need to operate simultaneously. Therefore, the input bandwidth of each memory array is maintained at 64 bytes, and this 64-byte bandwidth is time-division multiplexed to access data from ITU 915 and ALU 917. The RAM 916 also has two output ports, one connected to the RDMA 906 and the other to the ALU 917. After the ALU 917 completes its calculation, the RAM 916 sends the calculation result to the DRAM 604 via the RDMA 906. Therefore, time-division multiplexing is also used at the output end, utilizing the 64-byte output bandwidth to achieve data transmission to the RDMA 906 and ALU 917.

[0162] The present invention also discloses a computer-readable storage medium having computer program code stored thereon, wherein when the computer program code is run by the processing device 603, the aforementioned preprocessing method is executed.

[0163] This invention utilizes the characteristics of the Winograd algorithm in its hardware design to achieve accelerated versatility. It proposes a pipelined operation method to speed up Winograd convolution operations and fully leverages reusable resources through time-division multiplexing and broadcast routing during hardware implementation. The preprocessing scheme proposed in this invention is compatible with the Winograd convolution algorithm, achieving the technical benefits of ensuring network accuracy, performance acceleration, area reduction, and power consumption reduction.

[0164] Depending on the application scenario, the electronic devices or apparatus of the present invention may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus of the present invention can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present invention can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0165] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this invention is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also refer to the relevant descriptions of other embodiments.

[0166] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0167] In this invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same position or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this invention. Additionally, in some scenarios, multiple units in the embodiments of this invention may be integrated into one unit or each unit may exist physically independently.

[0168] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0169] The foregoing can be better understood in accordance with the following terms:

[0170] Clause A1. A method for preprocessing neuron data and weight data of Winograd convolution, comprising: setting the size of a data unit as the product of the vectorization direction and the number of bits of data; and storing the neuron data in aligned units of the data units.

[0171] Clause A2, the method according to Clause A1, further includes: determining whether the input size of any dimension in the neuron data is odd; and if so, filling the dimensions with odd input sizes to make the input size even.

[0172] Clause A3, the method according to Clause A2, wherein if the input size of any dimension of the neuron data is not odd, the method further comprises: maintaining the dimension whose input size is not odd.

[0173] Clause A4, the method according to Clause A1 further includes: setting the size of the basal convolution; and converting the weight data into Winograd weight data according to the size of the basal convolution.

[0174] Clause A5, the method described in Clause A4, further includes: storing the Winograd weight data in aligned units of the data cells.

[0175] Clause A6. A computer-readable storage medium having stored thereon computer program code preprocessed neuron data and weight data of Winograd convolution, wherein when the computer program code is run by a processing device, it performs the method described in any one of Clauses A1 to A5.

[0176] Clause A7. A combined processing apparatus for preprocessing neuron data and weight data of Winograd convolution, comprising: a processing device for setting the size of a data unit as the product of the vectorization direction and the number of bits of data; and off-chip memory for storing the neuron data in units aligned to the data units.

[0177] Clause A8. The combined processing apparatus according to Clause A7, wherein if the processing apparatus determines that the input size of any dimension in the neuron data is odd, then the processing apparatus fills the dimension with the odd input size to make the input size even.

[0178] Clause A9. The combined processing apparatus according to Clause A8, wherein if the processing apparatus determines that the input size of any dimension of the neuron data is not odd, then the processing apparatus maintains the dimension where the input size is not odd.

[0179] Clause A10, the combined processing apparatus according to Clause A7, wherein the processing apparatus sets the size of the basal convolution and converts the weight data into Winograd weight data according to the size of the basal convolution.

[0180] Clause A11, the combined processing apparatus according to Clause A10, wherein the off-chip memory stores the Winograd weight data in units aligned to the data cells.

[0181] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1.A method for preprocessing neuron data and weight data of Winograd convolution, comprising: neuron data padding, ensuring that the padded neuron data can be completely divided into data units; base convolution instruction scheduling, setting the size of base convolution, splitting and selecting appropriate base convolution combination according to the size of convolution kernel, and converting original weight data into Winograd weight data according to the size of selected base convolution; The storage dimension reconstruction of the input data is performed, and the size of the data unit is set as the vectorization length along the vectorization direction l in units of the product of the data bit number and aligning the neuron data in units of the data units, aligning Winograd weight data in units of the data units, and storing the neuron data and Winograd weight data in units of the data units. 2.The method of claim 1, further comprising: determining whether the input size of any dimension of the neuron data is odd; and if so, padding the dimension with the odd input size to make the input size even. 3.The method of claim 2, wherein if the input size of any dimension of the neuron data is not odd, the method further comprises: maintaining the dimension with the non-odd input size. 4.A computer readable storage medium having stored thereon computer program codes for preprocessing neuron data and weight data of Winograd convolution, which, when executed by a processing device, perform the method of any one of claims 1 to 3. 5.A combined processing device for preprocessing neuron data and weight data of Winograd convolution, comprising: a processing device configured to perform the following operations: neuron data padding, ensuring that the padded neuron data can be completely divided into data units; base convolution instruction scheduling, setting the size of base convolution, splitting and selecting appropriate base convolution combination according to the size of convolution kernel, and converting original weight data into Winograd weight data according to the size of selected base convolution; The storage dimension reconstruction of the input data is performed, and the size of the data unit is set as the vectorization length along the vectorization direction l in units of the product of the data bit number aligning the neuron data in units of the data units, aligning Winograd weight data in units of the data units; an off-chip memory configured to store the neuron data and Winograd weight data in units of the data units. 6.The combined processing device of claim 5, wherein if the processing device determines that the input size of any dimension of the neuron data is odd, the processing device pads the dimension with the odd input size to make the input size even. 7.The combined processing device of claim 6, wherein if the processing device determines that the input size of any dimension of the neuron data is not odd, the processing device maintains the dimension with the non-odd input size.

Images