Cache for staging neuron data in winograd convolution, device and board

Abstract

The application relates to a cache for temporarily storing neuron data in Winograd convolution, a computing device, an integrated circuit device and a board card, wherein the cache comprises four storage arrays, each of the storage arrays comprises four storage blocks, and the scale of each of the storage blocks is d w-bit storage units, wherein d is the number of addresses in the storage units. The application has the technical effects of ensuring network precision, accelerating performance, reducing area and reducing power consumption.

Description

Technical Field

[0001] This invention generally relates to the field of neural networks. More specifically, this invention relates to a cache, computing device, integrated circuit device, and board for temporarily storing neuron data in Winograd convolutions. Background Technology

[0002] With the rapid development of the information age, research in the fields of artificial intelligence and machine learning is booming, and related industries are flourishing. Convolutional neural networks have a wide range of applications in computer vision, autonomous driving, machine translation, speech recognition, smart homes, and many other areas.

[0003] Convolutional neural networks have a large number of parameters and require a lot of computation, which severely limits their performance in portable mobile terminals with limited area and computing power. At the same time, non-dedicated processors also cause huge power consumption when performing convolution operations.

[0004] Winograd convolution is a method for accelerating convolution based on multinomial interpolation. It involves dividing the two inputs of the convolution operation—neurons and weights—by a certain scale, performing a linear transformation (Winograd forward transform) on each, then multiplying the transformed neurons and weights in pairs, and finally performing a linear transformation (Winograd inverse transform) on the result. The result is then equivalent to the original convolution operation.

[0005] Since the forward and inverse transformation matrices of neurons and weights in the Winograd convolution operation are composed of simple, fixed values, the forward and inverse transformation processes of Winograd neurons and weights can be implemented using only addition. The multiplication operations required in the Winograd algorithm only occur during positional multiplication, and the multiplication complexity of this process is significantly reduced compared to the original convolution algorithm. Because the hardware overhead (timing, power consumption, area) for implementing multiplication is much higher than implementing addition of the same bit width, replacing the original convolution operation with Winograd convolution brings significant benefits in hardware energy efficiency and computation time.

[0006] However, no hardware is currently designed specifically for accelerating the Winograd convolution algorithm, preventing existing AI chips from fully leveraging the advantages of Winograd convolution operations. Therefore, a hardware device capable of efficiently running the Winograd convolution algorithm is urgently needed. Summary of the Invention

[0007] To at least partially solve the technical problems mentioned in the background art, the present invention provides a cache, computing device, integrated circuit device, and board for temporarily storing neuron data in Winograd convolution.

[0008] In one aspect, the present invention discloses a cache for temporarily storing neuron data in a Winograd convolution, the cache comprising four storage arrays, each storage array comprising four storage blocks, each storage block being a storage unit of size d w bits, where d is the number of addresses in the storage unit.

[0009] In another aspect, the present invention discloses a computing device including the aforementioned cache.

[0010] In another aspect, the present invention discloses an integrated circuit device including the aforementioned computing device, and also discloses a board including the integrated circuit device according to the aforementioned.

[0011] The hardware structure proposed in this invention can match the Winograd convolution acceleration algorithm, and has the technical effects of ensuring network accuracy, performance acceleration, area reduction and power consumption reduction. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts wherein:

[0013] Figure 1 This is a schematic diagram illustrating the convolution operation between the convolution kernel and the input neuron image;

[0014] Figure 2 This is a schematic diagram showing the transformation of the original convolution of F(2×2,3×3) into a Winograd convolution;

[0015] Figure 3 This is a visual diagram illustrating the bitwise multiplication operation;

[0016] Figure 4 This is a schematic diagram illustrating the homogeneous operation between the positively transformed data and the weights;

[0017] Figure 5 This is a structural diagram of the board card according to an embodiment of the present invention;

[0018] Figure 6 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present invention;

[0019] Figure 7 This is a schematic diagram illustrating the internal structure of a computing device according to an embodiment of the present invention;

[0020] Figure 8 This is a schematic diagram showing the overlapping portion during a positive transformation;

[0021] Figure 9 This is a schematic diagram illustrating a neuron cache according to an embodiment of the present invention;

[0022] Figure 10 This is a schematic diagram illustrating the positive transformation unit according to an embodiment of the present invention;

[0023] Figure 11 This is a schematic diagram illustrating the positive transform data buffer according to an embodiment of the present invention;

[0024] Figure 12 This is a schematic diagram illustrating the weight cache of an embodiment of the present invention;

[0025] Figure 13 This is a schematic diagram illustrating the inverse transformation unit of an embodiment of the present invention; and

[0026] Figure 14 This is a schematic diagram illustrating the connection relationship of the result cache in an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0029] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0030] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0031] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0032] The Winograd convolution acceleration algorithm (hereinafter referred to as the Winograd algorithm or Winograd convolution) utilizes a linear transformation on the operands in convolution operations to find the transformation method that requires the fewest multiplications, and then replaces the necessary multiplication operations with some addition operations. From a hardware perspective, multipliers are more complex in structure, consume more power, and have worse overall processing performance than adders. In practice, the Winograd algorithm, which replaces multiplication with addition, has a significant advantage in processing convolution operations.

[0033] For two-dimensional convolution, assuming the input neuron image size is H×W (H is the height of the input neuron image, W is the width of the input neuron image), and the weights are r×s (r is the height of the weights, s is the width of the weights), the convolution result can be represented as F(m×n, r×s), where m×n is the size of the output neuron image, m is the height of the output neuron image, and n is the width of the output neuron image. In order to reduce hardware complexity, improve versatility, and achieve good acceleration, embodiments of the present invention use convolution kernels (i.e., weights) of no greater than 3×3 as base convolution units to perform Winograd convolution operations of arbitrary size with a stride of 1. Embodiments of the present invention decompose any F(m×n, r×s) into calculations of five types of base convolutions with operation sizes of 3×3, 3×2 (or 2×3), 3×1, 2×2, and 2×1, and then combine them. More specifically, embodiments of the present invention decompose any F(m×n, r×s) into a combination of basal convolution calculations of F(2×2, 3×3), F(2×2, 3×2), F(2×2, 2×3), F(2×2, 3×1), F(2×2, 2×2), and F(2×2, 2×1). It should be noted that since 1×1 convolution operations cannot be accelerated using Winograd convolution, the 1×1 size is not included in the basal convolution units defined in embodiments of the present invention.

[0034] Taking an input neuron image of size 6×6 and stride 1 as an example, F(2×2, 5×5), before performing Winograd convolution acceleration computation using the computing device of this embodiment, the 6×6 input neuron image and the 5×5 convolution kernel need to be linearly split based on the base convolution unit. The splitting process is as follows: Figure 1 As shown.

[0035] Figure 1The diagram illustrates a 5×5 convolution kernel 101 convolved with a 6×6 input neuron image 102 to obtain a 2×2 convolution result 103. The convolution kernel 101 needs to be split into sizes such as 3×3, 3×2 (or 2×3), 3×1, 2×2, and 2×1. In this embodiment, 3×3 is preferred, followed by 3×2 (or 2×3), then 3×1, then 2×2, and finally 2×1. According to this rule, convolution kernel 101 is split into four base convolution kernels: a 3×3 first base convolution kernel 104, a 3×2 second base convolution kernel 105, a 2×3 third base convolution kernel 106, and a 2×2 fourth base convolution kernel 107. That is, F(2×2, 5×5) is decomposed into one F(2×2, 3×3), one F(2×2, 3×2), one F(2×2, 2×3), and one F(2×2, 2×2). The input neuron image 102 is also correspondingly split into four sub-neuron data: a 4×4 first sub-neuron data 108, a 4×3 second sub-neuron data 109, a 3×4 third sub-neuron data 110, and a 3×3 fourth sub-neuron data 111.

[0036] Next, Winograd convolution operations are performed: the first base convolution kernel 104 is convolved with the first sub-neuron data 108 to produce the first sub-convolution result 112; the second base convolution kernel 105 is convolved with the second sub-neuron data 109 to produce the second sub-convolution result 113; the third base convolution kernel 106 is convolved with the third sub-neuron data 110 to produce the third sub-convolution result 114; and the fourth base convolution kernel 107 is convolved with the fourth sub-neuron data 111 to produce the fourth sub-convolution result 115.

[0037] Finally, the results of the first sub-convolution (112), the second sub-convolution (113), the third sub-convolution (114), and the fourth sub-convolution (115) are added together to obtain the convolution result 116, which is the same as the convolution result 103. This is an example of using the Winograd convolution algorithm to implement the original convolution operation.

[0038] Furthermore, the Winograd algorithm can be expressed by the following formula:

[0039] Y = A T [(GgG T )⊙(B T dB)]A

[0040] Where Y represents the output matrix of the convolution operation, and A T G is the inverse transformation multiplied by a constant matrix on the left, G is the weight transformation multiplied by a constant matrix on the left, and g is the weight of the original convolution. T The weight transformation is a right multiplication of a constant matrix, where ⊙ denotes positional multiplication, B TLet d be the left-multiplied constant matrix of the neuron transformation, d be the neuron data, B be the right-multiplied constant matrix of the neuron transformation, and A be the right-multiplied constant matrix of the inverse transformation. The left-multiplied and right-multiplied matrices of each transformation are simply transposes.

[0041] Taking F(2×2, 3×3) as an example, the aforementioned constant matrices are shown below:

[0042]

[0043]

[0044]

[0045] Figure 2 The diagram illustrates the transformation of the original convolution of F(2×2, 3×3) into a Winograd convolution. As shown, neuron data 201 is convolved with convolution kernel 202. During computation, neuron data 201 is arranged in a row according to the elements in sliding window 203. Sliding window 203 slides four times to form a 4×9 matrix 204. Then, the elements of convolution kernel 202 are arranged in a column to form a 9×1 matrix 205. The 4×9 matrix 204 and the 9×1 matrix 205 are convolved to obtain a 4×1 convolution result 206.

[0046] Next, based on the dotted lines in the diagram, the 4×9 matrix 204 is transformed into a 2×3 matrix 207, the 9×1 matrix 205 into a 3×1 matrix 208, and the 4×1 convolution result 206 into a 2×1 convolution result 209. After the linear transformation, the first element of the 2×1 convolution result 209 is R0 = M0 + M1 + M2, and R1 = M1 - M2 - M3. M0, M1, M2, and M3 can be represented by the following formula:

[0047] M0 = (K0 - K2)·W0

[0048] M3=(K1-K3)·W2

[0049] Through the aforementioned segmentation and linear transformation, the original convolution operation involved 36 multiplications, while the Winograd algorithm only requires 16 multiplications, reducing the computational complexity of multiplication by 2.25 times.

[0050] From the above transformation of the 2D convolution using the Winograd algorithm, it can be seen that the Winograd algorithm mainly consists of the following steps. First, the weights are multiplied by the weight constant matrix on the left and right, i.e., GgG TWe obtain the weights after the Winograd linear transformation, i.e., the Winograd weights; then, we perform a forward transformation operation on the neuron data, which is the left and right multiplication of the neuron constant matrix, i.e., B. T dB, obtaining the forward transform data after the Winograd linear transform. Then, the forward transform data and the Winograd weight matrix are multiplied in pairs, i.e., (GgG T )⊙(B T dB) is used to obtain the positional multiplication data. Finally, the positional multiplication data is subjected to an inverse transformation operation, which is the left and right multiplication of the Winograd inverse transformation constant matrix, i.e., A T LA, where L is [(GgG T )⊙(B T This process, using dB), ultimately yields a convolution result equivalent to the original convolution.

[0051] From a hardware design perspective, the embodiments of this invention, taking into account the dependencies and operational differences among the three processes described above, streamline these three major transformation steps to achieve more efficient acceleration performance. The design of the forward transformation operation, the positional multiplication operation, and the inverse transformation operation will be described separately below.

[0052] Embodiments of the present invention utilize a forward transform unit to implement the forward transform operation, that is, to execute B. T dB, according to the rules of Winograd convolution, is a forward transform multiplied by matrix B on the left. T The size of the matrix is ​​(m+r-1)×(m+r-1), and the size of the right-multiplied matrix B is (n+s-1)×(n+s-1). Since the forward transformation is left-multiplied by matrix B... T Since the elements of the matrix B being multiplied by the right matrix consist of 0, 1, and -1, the matrix multiplication operation of the forward transformation can be decomposed into a fixed-pattern addition operation. The computing device in this embodiment of the invention is configured with a specific number of floating-point adders to complete the linear addition operations required for the entire matrix multiplication. Because this embodiment of the invention converts any original convolution into a basal convolution for calculation, the size of the forward transformation unit is related to the operation of the aforementioned five basal convolution sizes. Therefore, the following will use a 2×2 convolution result as an example (i.e., m = n = 2) to explain the data for the aforementioned five basal convolution operations FP32.

[0053] Taking a 3×3 base convolution as an example, its It can be represented as:

[0054]

[0055] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit directly corresponds to the number of adders, which is 4×(n+s-1)+4×(m+r-1)=32flops (floating-point operations per second). The input and output quantities of the forward transformation unit are (r+1)(s+1)×32=16×32 bits, respectively. The reason for multiplying the formula by 32 bits is for FP32 data, which is a 32-bit sequence. When the input and output quantities of the forward transformation unit are the same as the operation time, the hardware utilization of the forward transformation unit is most ideal. Therefore, the ratio of the input / output bandwidth of the forward transformation unit to the addition operation is preferably 16:32=1:2. In other words, when the cache bandwidth (or vectorization length) is l, the input bandwidth and output bandwidth of the forward transformation unit are l×32 bits, and the computing power of the adder group of the forward transformation unit is 2×l flops. Each operation produces 16 final results. Considering that 8 intermediate results will be generated during the operation, the minimum number of registers in the register file is 1×32×(16+8).

[0056] Taking a 3×2 base convolution as an example, its It can be represented as:

[0057]

[0058] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit is 4×(n+s-1)+2×(m+r-1)=20flops, while the input and output quantities of the forward transformation unit are (r+1)(s+1)×32=12×32 bits respectively. To improve the hardware utilization of the forward transformation unit, the optimal ratio of the input / output bandwidth to the addition operation is 12:20=3:5. That is, the input bandwidth and output bandwidth are 1×32 bits, and the computing power of the adder group is… Each calculation produces 12 final results and 6 intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(12+6).

[0059] Taking a 2×2 base convolution as an example, its It can be represented as:

[0060]

[0061] Based on the above formula, the forward transformation computing power requirement of the forward transformation unit is 2×(n+s-1)+2×(m+r-1)=12flops. The input and output quantities of the forward transformation unit are (r+1)(s+1)×32=9×32 bits each. Therefore, the optimal ratio of the input / output bandwidth of the forward transformation unit to the addition operation is 9:12=3:4. That is, the input bandwidth and output bandwidth are 1×32 bits, while the computing power of the adder group is… Each calculation produces 9 final results and 6 intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(9+6).

[0062] Taking a 3×1 base convolution as an example, its It can be represented as:

[0063]

[0064] Based on the above formula, the forward transformation unit requires 4 flops of computing power, while the input and output quantities of the forward transformation unit are (r+1)×32=4×32 bits each. Therefore, the optimal ratio of the input / output bandwidth of the forward transformation unit to the addition operation is 4:4=1:1. That is, the input and output bandwidths are l×32 bits, and the computing power of the adder group is l flops. Each calculation produces 4 final results and 2 intermediate results. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is l×32×(4+2).

[0065] Taking a 2×1 base convolution as an example, its It can be represented as:

[0066]

[0067] Based on the above formula, the forward transformation unit's computational power requirement is 2 flops. The input and output quantities of the forward transformation unit are (r+1)×32=3×32 bits each. Therefore, the optimal ratio of the input / output bandwidth to the addition operation is 3:2. That is, the input and output bandwidths are 1×32 bits, while the computational power of the adder group is... Each calculation produces 3 final results and 1 intermediate result. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(3+1).

[0068] To simultaneously satisfy and support the aforementioned five types of base convolution operations, the embodiments of the present invention select that the input bandwidth and output bandwidth of the forward transform unit are the same, and the computing power of the addition operation is twice that of the input bandwidth and output bandwidth, that is, the input bandwidth and output bandwidth are both 1×32 bits, while the computing power of the adder group is 2×1 flops, and the number of register files is 1×32×(16+8).

[0069] Next, consider the positional multiplication and accumulation operator. Based on a comprehensive consideration of hardware design, scheduling strategy and execution performance, the embodiments of the present invention combine the positional multiplication operation and the feature map direction of the convolutional neuron data. By using the same positional multiplication and accumulation operator, it can not only effectively reduce the overall complexity of hardware design and resource consumption, but also reduce the number of on-chip cache accesses, save power consumption and area, and improve performance.

[0070] Assume the parameters of the convolutional layer are: input batch size N, input neuron channel size Ci, input neuron height Hi, input neuron width Wi, output neuron channel size Co, output neuron height Ho, output neuron width Wo, kernel size r×s, and stride 1. Since this embodiment supports F(2×2, r×s) operations, Ho = Hi - r + 1, Wo = Wi - s + 1, and the number of Winograd computational units... Where T is the number of blocks along the HW direction.

[0071] Due to the limited on-chip cache capacity, the computing device in this embodiment performs calculations in a single batch (N=1). Therefore, the scale of the input neuron data input to the computing device is [1 Ci Hi Wi], the scale of the forward transformation data is [1 Ci T(r+1)×(S+1)], the scale of the original weights is [Co Ci rs], and the scale of the Winograd weights is [1 Co Ci(r+1)×(S+1)].

[0072] Figure 3 The diagram illustrates the aforementioned positional multiplication operation. Since N=1, the aforementioned data can be reduced to three dimensions. The size of the forward transformation data 301 is [Ci T(r+1)×(S+1)], and its three dimensions are Ci, T (i.e., the number of HW blocks) and (r+1)×(S+1), respectively. The Winograd weight 302 is [Co Ci (r+1)×(S+1)], and its three dimensions are Co, Ci and (r+1)×(S+1), respectively. The positional multiplication operation involves cross-multiplying Co in the HW direction and accumulating in the Ci direction to obtain the positional multiplication data 303, which has a size of [Co T(r+1)×(S+1)], and its three dimensions are Co, T and (r+1)×(S+1), respectively.

[0073] More specifically, the forward transformation data 301 involves multiplying T [Ci(r+1)×(S+1)] data units bitwise, and the Winograd weights 302 involve multiplying Co [Ci(r+1)×(S+1)] data units bitwise to obtain an intermediate result of [Ci(r+1)×(S+1)]. Then, the result is accumulated along the Ci direction. This process is identical to matrix multiplication, thus it can be combined into a single matrix multiplication operation, making more efficient use of hardware resources and reducing the consumption of intermediate storage register resources.

[0074] Since the forward transform data 301 includes T data units of [Ci(r+1)×(S+1)], and the Winograd weights 302 include Co data units of [Ci(r+1)×(S+1)], each data unit of the forward transform data 301 needs to be multiplied by each data unit of the Winograd weights 302. For example... Figure 4 As shown, during the positional multiplication operation, one data unit 401 of the forward transformation data 301 is homogeneously operated on with Co weight data units, i.e., the Co direction is used as the direction of parallel computation, and an intermediate result 402 is produced. Then, the next data unit is taken from the forward transformation data 301 and homogeneously operated on with Co weight data units again, producing the next intermediate result. This process is repeated until all T data units have been calculated, thus obtaining the positional multiplication data 303.

[0075] When the aforementioned data units are multiplied in pairs and accumulated along the feature map direction, the required computation is (Ci + Ci-1) × (r+1) × (S+1) floats. Since the value of Ci is often very large, it is difficult to use it as the granularity input to the pair multiplication and accumulation operator in practice. Therefore, this embodiment can further split Ci, perform multiplication and accumulation operations in units of vectorized length l, and split the multiplication and accumulation operation of the other dimension (r+1) × (S+1) into (r+1) × (S+1) steps to be completed sequentially. Finally, all results are added along the Ci direction to obtain the final result.

[0076] Since the output bandwidth of the forward transform unit is l×32 bits, to ensure that the overall pipeline time from the forward transform unit to the bitwise multiply-accumulate unit is the same, in this embodiment, the computing power of each bitwise multiply-accumulate unit in the bitwise multiply-accumulate unit is set to l+(l-1)flops, which includes l multiplication operations and l-1 addition operations. If the bitwise multiply-accumulate unit has ω parallel dimensions, that is, including ω bitwise multiply-accumulate units operating simultaneously, then the computing power of the bitwise multiply-accumulate unit is ω×(l+(l-1))flops, which is a function of ω and l.

[0077] This embodiment also includes an inverse transformation unit to perform the inverse transformation operation, which involves left-multiplying the matrix according to the inverse transformation. and right multiplication of matrix A (n+s-1)×2 Perform A T LA calculation, where L is (GgG T )⊙(B T dB). Due to the inverse transformation, the matrix A is multiplied on the left. T Since the elements of the right-multiplied matrix A are also composed of 0, 1, and -1, the matrix multiplication operation of the inverse transform can also be decomposed into a fixed-pattern addition operation. The adder group of the inverse transform unit is configured with a specific number of floating-point adders to complete the linear addition operations required for the entire matrix multiplication. The following explanation also considers the five types of base convolution to determine the size of the inverse transform unit.

[0078] Taking a 3×3 base convolution as an example, its It can be represented as:

[0079]

[0080] Based on the above formula, the inverse transform computing power of ITU 715 is 24 flops, while the input bandwidth is (r+1)(s+1)×32=16×32 bits, and the output bandwidth is (s+1)×32=4×32 bits. Similarly, when the input bandwidth and computing power of the inverse transform unit are the same, the hardware utilization of the inverse transform unit is optimal. Therefore, the optimal ratio of input bandwidth to addition operation is 16:24=2:3, that is, the input bandwidth is 1×32 bits, and the computing power of the adder group is… Each calculation produces 16 final results, with no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×16.

[0081] Taking a 3×2 base convolution as an example, its It can be represented as:

[0082]

[0083] Based on the above formula, the inverse transform unit has an inverse transform computing power of 16 flops, an input bandwidth of 12 × 32 bits, and an output bandwidth of 4 × 32 bits. The optimal ratio of input bandwidth to addition operation is 12:16 = 3:4, meaning the input bandwidth is 1 × 32 bits, while the computing power of the adder group is... Each calculation produces 12 final results, with no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×12.

[0084] Taking a 2×2 base convolution as an example, its It can be represented as:

[0085]

[0086] Based on the above formula, the inverse transform unit has an inverse transform computing power of 10 flops, an input bandwidth of 9 × 32 bits, and an output bandwidth of 4 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 9:10, meaning the input bandwidth is 1 × 32 bits, while the computing power of the adder group is... Each calculation produces 9 final results, with no intermediate results. To maximize the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×9.

[0087] Taking a 3×1 base convolution as an example, its It can be represented as:

[0088]

[0089] Based on the above formula, the inverse transformation unit has an inverse transformation computing power of 4 flops, an input bandwidth of 4 × 32 bits, and an output bandwidth of 2 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 4:4 = 1:1, that is, the input bandwidth is l × 32 bits, and the computing power of the adder group is l flops. Each calculation will produce 4 final results and 2 intermediate results. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is l × 32 × (4 + 2).

[0090] Taking a 2×1 base convolution as an example, its It can be represented as:

[0091]

[0092] Based on the above formula, the inverse transform unit has an inverse transform computing power of 2 flops, while the input bandwidth is 3 × 32 bits and the output bandwidth is 3 × 32 bits. Therefore, the optimal ratio of input bandwidth to addition operation is 3:2, i.e., the input bandwidth is 1 × 32 bits, and the computing power of the adder group is... Each calculation produces 3 final results and 1 intermediate result. Under the premise of maximizing the use of the register file in the pipeline, the minimum number of registers in the register file is 1×32×(3+1).

[0093] To simultaneously satisfy and support the aforementioned five types of base convolution operations, the computational power of the addition operation in the inverse transform unit can be set to a value equal to the input bandwidth. This means that when the input bandwidth is 1×32 bits, the computing power of the adder group is [number] times ...

[0094] However, to simplify the hardware design, this embodiment can further consider having the same hardware configuration for the forward and inverse transform units. While simultaneously meeting the requirements of both the forward and inverse transform units, this embodiment chooses to use the forward transform unit design for the inverse transform unit, meaning the input and output bandwidths are the same, and the computational power of the addition operation is twice that of the input and output bandwidths. In other words, the inverse transform unit has an input bandwidth of 1×32 bits and an output bandwidth of 1×32 bits, resulting in a computational power of 2×1 flops for the adder group.

[0095] In summary, the bandwidth and computing power of the three core modules (forward transform unit, bitwise multiplication accumulator, and inverse transform unit) performing Winograd convolution operations in this embodiment have been matched. That is, the input bandwidth of each of the three core modules is set to 1×32 bits, and the output bandwidth is also set to 1×32 bits. The computing power of the forward transform unit is 2×1 flops, the computing power of the bitwise multiplication accumulator is ω×(1+(1-1)) flops, and the computing power of the inverse transform unit is 2×1 flops.

[0096] As mentioned above, the Winograd convolution operation is directly related to the vectorization length parameter *l*. The vectorization length parameter *l* is the minimum processing length and relates to the reuse of neuron transformations in this embodiment's computing device. A larger *l* parameter results in a higher reuse rate, while the required memory access, computational load, power consumption, and average hardware design area decrease proportionally. However, the parameters of the neural network convolutional layers change with the network model. As the vectorization length parameter *l* increases, when some network models have fewer channels than *l*, computational power is wasted, affecting the acceleration effect and leading to additional overhead in area and power consumption. Therefore, a trade-off analysis of these two factors is necessary when determining the vectorization length *l* to plan the most suitable vectorization length parameter configuration.

[0097] Based on empirical values, weights were assigned to several key hardware components in this embodiment (such as the FP32 adder, bitwise multiplication unit, registers, etc.) to obtain their computational power and resource overhead functions. It was found that when l is greater than 16, the utilization rate of hardware resources can be guaranteed to be at a high level. Furthermore, the number of input and output channels of commonly used neural network models (such as LeNet, VGG16, VGG19, AlexNet) were considered, and their computational power loss was calculated. It was found that when l is greater than 64, the overall computational power loss will increase significantly. From the above two quantitative analyses, it can be concluded that when the vectorization length parameter l is between 16 and 64, the computing device of this embodiment performs well. If versatility is further considered to meet possible future network structures and parameters, this embodiment preferably chooses l = 16.

[0098] Figure 5A schematic diagram of the structure of the aforementioned embodiment, presented in the form of a circuit board, is shown. Figure 5 As shown, board 50 includes chip 501, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 50 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0099] Chip 501 is connected to external device 503 via external interface device 502. External device 503 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 503 to chip 501 via external interface device 502. The calculation results from chip 501 can be transmitted back to external device 503 via external interface device 502. Depending on the application scenario, external interface device 502 may have different interface forms, such as a PCIe interface.

[0100] The board 50 also includes a storage device 504 for storing data, which includes one or more memory cells 505. The storage device 504 is connected to and transmits data with the controller 506 and the chip 501 via a bus. The controller 506 in the board 50 is configured to regulate the state of the chip 501. Therefore, in one application scenario, the controller 506 may include a microcontroller (MCU).

[0101] Figure 6 This is a structural diagram illustrating the combined processing device in chip 501 of this embodiment. (As shown...) Figure 6 As shown, the combined processing device 60 includes a computing device 601, an interface device 602, a processing device 603, and a DRAM 604.

[0102] The computing device 601 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor, to perform deep learning or machine learning calculations, especially Winograd convolution operations. It can interact with the processing device 603 through the interface device 602 to jointly complete the user-specified operations.

[0103] Interface device 602 is used to transmit data and control commands between computing device 601 and processing device 603. For example, computing device 601 can obtain input data from processing device 603 via interface device 602 and write it into on-chip cache of computing device 601. Further, computing device 601 can obtain control commands from processing device 603 via interface device 602 and also write them into on-chip cache of computing device 601. Alternatively or optionally, interface device 602 can also read data from on-chip cache of computing device 601 and transmit it to processing device 603.

[0104] The processing device 603, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 601. Depending on the implementation, the processing device 603 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 601 of this invention can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 601 and the processing device 603 are considered together, they are considered to form a heterogeneous multi-core structure.

[0105] DRAM 604 is used to store data to be processed. It is off-chip memory, typically 16GB or larger in size, and is used to store data from computing device 601 and / or processing device 603, particularly storing neuron data and weights to be used for Winograd convolution operations. In this embodiment, processing device 603 has already linearly transformed the original convolution weights into Winograd weights GgG in advance. T It is stored in DRAM 604.

[0106] Figure 7A structural diagram of computing device 601 is shown. Computing device 601 includes a bus 701, a direct memory access (DMA) module 702, an instruction cache (IRAM) 707, a decoding unit (IDU) 708, a neuron cache (NRAM) 709, a forward transformation unit (NTU) 710, a forward transformation data cache (WNRAM) 711, a weight cache (WRAM) 712, a positional multiplication accumulator (MAC) 713, a positional multiplication data cache (WRRAM) 714, an inverse transformation unit (ITU) 715, a result cache (RRAM) 716, and an arithmetic logic unit (ALU) 717.

[0107] Bus 701 is a common communication trunk line for transmitting information between devices. It is a transmission line bundle composed of wires. According to the types of information transmitted by the combined processing device 60, bus 701 is a collective term for data bus, address bus, and control bus, used to transmit data, data address, and instructions respectively. Bus 701 serves as the communication channel between DRAM 604 and computing device 601, and in this embodiment, it is specifically PCIe.

[0108] The DMA module 702 is used to copy data from one address space to another, typically moving data between external memory (such as DRAM 604) and the internal cache of the computing device 601. During DMA transfer, the processing device 603 relinquishes bus control to the DMA module 702, which controls the bus 701 to perform data transfer. After the DMA transfer is complete, the DMA module 702 returns bus control to the processing device 603.

[0109] DMA module 702 includes Neuron Direct Memory Access (NDMA) 703, Weight Direct Memory Access (WDMA) 704, Instruction Direct Memory Access (IDMA) 705, and Result Direct Memory Access (RDMA) 706. NDMA 703 is used to input neuron data from DRAM 604, WDMA 704 is used to input Winograd weights from DRAM 604, IDMA 705 is used to input instructions from DRAM 604, and RDMA 706 is used to output calculation results to DRAM 604. In other embodiments, NDMA 703, WDMA 704, IDMA 705, and RDMA 706 can be implemented using the same direct memory access module.

[0110] The IRAM 707 is used to temporarily store instructions input from the IDMA 705. The IDU 708 fetches instructions from the IRAM 707, decodes them, and controls the operation of other units based on the decoded instructions. The IDU 708 is the decoding and scheduling unit of the entire computing device 601. It is responsible for decoding the control instructions obtained from the DRAM 604, converting them into control signals to coordinate the operation of various modules / units on the chip, and is also responsible for instruction order preservation, dependency removal, branch prediction, exception handling, interrupt handling, and many other tasks. Figure 7 Thin arrows indicate control flow, while thick arrows indicate data flow.

[0111] Since the computing device 601 is primarily designed for Winograd convolution calculations and lacks general-purpose processing capabilities or has limited general-purpose processing capabilities, it relies heavily on the scheduling and data communication of the processing device 603 during task execution. This results in frequent input / output communication between the computing device 601 and the processing device 603, significantly limiting the performance of the computing device 601. To address this, the computing device 601 incorporates multiple small-capacity on-chip caches to buffer reusable temporary data, such as NRAM 709, WNRAM 711, WRAM 712, and WRRAM 714.

[0112] During on-chip and off-chip data transfer, neuron data and Winograd weights are transferred in single-batch (N=1) units. Specifically, the data unit for neuron data is [Ci Hi Wi], while the data unit for Winograd weights is [Co Ci(r+1)×(s+1)]. The result after the Winograd convolution operation is [Co HoWo]. The former two are input data, and the latter is output data. These represent the minimum throughput for transmission and computation in computing device 601. The actual data throughput depends on the size of the on-chip cache and the computation scheduling process, which will be further explained below.

[0113] Based on the characteristics of convolution operations, it is known that the convolution operations involved in input data of the above scale can be split in multiple dimensions, such as in the Ci direction, the HW image direction, or the Co direction. However, when Winograd transformation is involved, the smallest operational splitting unit is F(2×2, r×s), and the smallest splitting unit in the HW direction is (r+1)×(s+1). Considering that the base convolution size for Winograd acceleration implemented by the computing device 601 does not exceed 3×3, this embodiment estimates the cache capacity based on the 3×3 base convolution, which consumes the most on-chip cache resources.

[0114] According to the rules of Winograd convolution, during the forward transformation operation, parallel processing is required in the Ci direction in units of vectorized length parameter l. During the positional multiplication and accumulation operation, parallel processing is required in the Co direction in units of l. During the inverse transformation, parallel processing is required in the Co direction in units of l. Therefore, the minimum input data block size of the neuron participating in the operation can be deduced to be [l(r+1)×(s+1)]. Since it is estimated using 3×3 radix convolution, the size of the neuron transformation result data block is [l 4×4]. The size of the Winograd weight data block that needs to be multiplied and accumulated with it is [l1 4×4]. The size of the positional multiplication output data block is [l 4×4]. The size of the inverse transformation output result is [l 2×2].

[0115] While designing an on-chip cache at this scale can meet all requirements and takes into account the design principles of reuse and low power consumption, the above-mentioned data only represents the minimum input / output storage data size for implementing the functionality. Further optimization potential for the input / output volume of Winograd convolution operations needs to be considered. This embodiment further plans the cache as follows.

[0116] During the positive transformation of the neuron, the operation is based on F(2×2, r×s) and l as the smallest implementation unit of vectorization length. The size of the data block retrieved each time is [l 4 4], and the step size of the neuron in retrieving data is kept at 2. For example Figure 8 As shown, there is a quarter-overlapping portion 806 between the data unit 801 to be transformed and the four data blocks 802, 803, 804, and 805 generated by the sliding window. The size of the overlapping portion 806 is [l 4 4]. As can be seen from the figure, during the forward transformation of the data unit 801, each of the data blocks 802, 803, 804, and 805 includes one overlapping portion 806, thus generating a total of four overlapping portions 806. When the data is moved by splitting it according to the smallest data unit [l 4 4], the data throughput required for the overlapping portion 806 increases fourfold, resulting in increased redundant data. To solve this problem, this embodiment further reduces the input / output by caching larger data units on the on-chip cache of the computing device 601.

[0117] As mentioned earlier, neuron data of size [Ci Hi Wi] is convolved with Winograd weights of size [Co Ci(r+1)×(s+1)]. This embodiment retains as many Winograd weights as possible on-chip, i.e., temporarily stores as many on-chip weights as possible. With [ll(r+1)×(s+1)], a batch of neuron data can be calculated with only one weight loading operation, thus saving the amount of weight data input / output.

[0118] For the output data, since the convolutional neural network also involves other network layer operations such as activation, pooling, and normalization, the convolution results need to be cached on-chip for subsequent network layer operations. Therefore, the computing device 601 reserves a fixed-capacity cache to store the convolution results. This cached data can share the cache space with the final results after various other layer operations, thus reducing the data throughput of reloading the convolution results for other layer operations and transmitting the computation results.

[0119] The above optimization analysis shows that the cache capacity for neuron data should be as large as possible to reduce the total throughput of neuron data. Since neuron data is accumulated along the Ci direction, the larger the amount of data stored along the Ci direction, the fewer times reloading and re-accumulation are required. Furthermore, the cache space for Winograd weights also needs to be as large as possible. Finally, this embodiment also needs to reserve corresponding output result space for other layer operations. In summary, this embodiment divides the on-chip cache into three main parts, each responsible for different functions: Nram 709 stores neuron data, Wram 712 stores Winograd weights, and Rram 716 stores convolution results. The computing device 601 further sets up two caches to temporarily store intermediate results: WNram 711 temporarily stores the data after forward transformation, and WRram 714 temporarily stores the data after bitwise multiplication and accumulation.

[0120] While a larger cache capacity for storing neuron data, Winograd weights, and convolution results is always better, the cache size is closely related to the configuration of the arithmetic unit resources. If the cache is configured too large, it will reduce the computing power of the computing device 601. The criterion for judgment is the balance between input / output bottleneck pressure and computational power pressure. In this embodiment, the size of NRAM 709 is set to... Where α is The direction factor is α, where β is the direction factor of HW; the size of Wram 712 is set to α×γ×[11 4 4], where γ is the direction factor of HW. The direction coefficients; the size of Rram 716 is set to β×γ×[l 2 2]. The time required to complete the calculation of these sizes of data is l×α×β×γ.

[0121] Preferably, in this embodiment, l is chosen to be 16, α to be 4, β to be 64, and γ to be 16. Considering that the data size of each FP32 is 4B, the storage capacity of the NRAM 709 storage array is... The storage capacity of Wram 712 is α×γ×[l 4 4]×4B=1MB, and the storage capacity of Rram 716 is β×γ×[l 2 2]×4B=256KB.

[0122] Back Figure 7 The NRAM 709, based on the decoded instructions, temporarily stores the neuron data sent by the NDMA 703. The NTU710, based on the decoded instructions, reads the neuron data from the NRAM 709 and performs a forward transform, i.e., a B-mode transformation. T The dB calculation is used to generate positive transformation data, which is then temporarily stored in WNram 711. Figure 9 A schematic diagram of the NRAM 709 is shown. In this embodiment, the NRAM 709 includes four memory arrays 901, 902, 903, and 904. Each memory array includes four memory blocks 905, 906, 907, and 908. Each memory block has a size of d w-bit storage cells, where d represents the number of addresses in the storage cell. Preferably, w is 128 and d is 1024, then the size of each storage block is 16KB, the size of each memory array is 64KB, and the total storage capacity of the NRAM 709 is 256KB. Its total width is 4 × w = 64B, and its depth is 4 × d = 4 × 1024.

[0123] In the width direction, the input bandwidth of the NRAM 709 is set to 4 bits, while the output bandwidth must match the input bandwidth of the NTU 710. As mentioned earlier, the input bandwidth of the NTU 710 is set to 1 × 32 bits, and 1 is preferably 16, so the input bandwidth of the NTU 710 is 64 bits. Therefore, the output bandwidth of the NRAM 709 is also 4 × w = 64 bits. The input and output of the NRAM 709 need to be performed simultaneously, so a dual-port design with both input and output is adopted.

[0124] Figure 10 A schematic diagram of the NTU 710 is shown. The NTU 710 includes an input buffer 1001, a register file 1002, an adder group 1003, and an output buffer 1004.

[0125] When the NTU 710 receives an instruction to load neuron data from the NRAM 709, the input buffer 1001 acts as a first-in-first-out queue buffer to temporarily store neuron data based on the input bandwidth of 64B. The neuron data loading phase continues until all data reception is complete, and the entire process is controlled by instructions sent by the IDU 708.

[0126] According to the decoded instructions, register file 1002 retrieves the temporarily stored neuron data from input buffer 1001 and stores it in a specific address in register file 1002 in the planned order of operations. This neuron data stored in a specific address in register file 1002 becomes the addition operand. In this embodiment, since the pipeline time lengths of the input, operation, and output stages of the NTU 710 should be equal, a cache hardware resource dependency phenomenon will occur. To solve the resource dependency problem, the register file 1002 is divided into two storage units of the same size: ping storage unit 1005 and pong storage unit 1006. The i-th addition operand and the resulting positive transformation data are temporarily stored in ping storage unit 1005, the (i+1)-th addition operand and the (i+1)-th positive transformation data are temporarily stored in pong storage unit 1006, and the (i+5)-th addition operand and the (i+5)-th positive transformation data are temporarily stored in ping storage unit 1005, overwriting the i-th addition operand and the (i+5)-th positive transformation data. The register file 1002 stores data according to this rule.

[0127] Adder group 1003 reads addition operands sequentially from specific addresses in register file 1002 according to the decoded instructions and performs addition operations. In this embodiment, there are two groups of adder groups 1003 to correspond to the addition operation scheduling direction. Each group includes 16 adders to correspond to the vectorization direction l. Each adder is an FP32 adder, which performs addition operations in the forward transformation of Winograd convolution in a specific order along the channel direction of neuron data. This specific order is to first calculate the left multiplication matrix B of the Winograd convolution. T The addition is performed, followed by the addition of the right multiplication matrix B of the Winograd convolution, finally generating the forward transform data, which is then stored back into register file 1002. The order of operations, register allocation, and operation time are all related to the size of the convolution filter and are controlled by instructions sent by IDU 708. This operation stage is data-dependent with the aforementioned stage of loading neuron data, and is executed in a pipelined manner, implemented by hardware counting.

[0128] Output buffer 1004 is also a first-in-first-out queue buffer, used to temporarily store forward transformation data sequentially from ping storage unit 1005 and ping storage unit 1006. This output stage depends on the overall completion of the computation stage in order to output the corresponding buffered data based on the output bandwidth 64B.

[0129] To save overhead, the forward transformation data needs to be multiplexed, so a WNram 711 is configured for temporary storage and repeated transmission. It includes multiple buffer units. Figure 11A schematic diagram of an exemplary WNram 711 is shown. As shown, the WNram 711 includes four cache units: a first cache unit 1101, a second cache unit 1102, a third cache unit 1103, and a fourth cache unit 1104. Forward transformation data from the NTU 710 is sent to one or more of these cache units via routing.

[0130] The WNram 711 sends forward transform data to the MAC 713 in a specific order for subsequent operations. The WNram 711 is designed to buffer a portion of the forward transform data, sending this data to the MAC 713 before storing the next portion of forward transform data, thus reducing the size of the WNram 711 through pipelining. Furthermore, considering that the forward transform data will be multiplied bitwise with Winograd weights of size γ × [l l4 × 4], the WNram 711 is configured to be sent to the MAC 713 in units of γ data blocks for computation. In this way, forward transform data is only output on average every γ cycles, effectively reducing the power consumption of the WNram 711. Accordingly, the first γ forward transform data are sequentially overwritten by the next γ forward transform data, thus limiting the minimum storage size of the WNram 711 to [l(r+1)(s+1)] × 4B, which is [l 4 4] × 4B = 1KB as mentioned earlier.

[0131] Specifically, the width of the first cache unit 1101, the second cache unit 1102, the third cache unit 1103, and the fourth cache unit 1104 is w1 bytes, and the depth is d1, and they are divided into m parts in the depth direction. In this embodiment, m is preferably 8, w1 is 64, and d1 is 128, so the width of each cache unit is 64 bytes, the depth is 128, and the address space is divided into 8 parts in the depth direction for data reuse. The size of each cache unit is 8KB, that is, the total capacity of WNram 711 is set to 32KB.

[0132] Back Figure 7 Wram 712 temporarily stores the Winograd weights sent by WDMA 704 according to the decoded instructions. MAC 713 reads the Winograd weights from Wram 712 and the forward transform data from WNram 711 according to the decoded instructions, and performs bitwise multiplication and accumulation operations on the forward transform data and the Winograd weights, that is, performs [(GgG T )⊙(B T The operation of dB)] is performed to generate the bitwise multiplication data, and the bitwise multiplication data is temporarily stored in WRram 714.

[0133] Figure 12A schematic diagram of Wram 712 is shown. In this embodiment, Wram 712 includes four storage arrays 1201, 1202, 1203, and 1204. WDMA 704 sends Winograd weights to storage arrays 1201, 1202, 1203, and 1204 via routing. Each storage array includes four storage blocks 1205, 1206, 1207, and 1208, and each storage block includes four storage cells 1209, 1210, 1211, and 1212. The size of each storage cell is 4×d×w. As mentioned earlier, w is 128 and d is 1024, so the size of each storage block is 64KB, and the size of each storage array is 256KB. The total capacity of Wram 712 is 1MB. For each storage block, its width is 4×w=512 bits. It is divided into 4 independent address storage spaces in the depth direction, each segment with a depth of d=1024, and the total depth is 4×d=4096.

[0134] In this embodiment, each memory array 1201, 1202, 1203, and 1204 independently has an input bandwidth and output bandwidth of 4×w B, and the total output bandwidth of Wram 712 is 4×4×w B. Specifically, when w is 128, the input bandwidth and output bandwidth of each memory array are 64B, and the total output bandwidth and total output bandwidth are 256B.

[0135] In this embodiment, the MAC 713 includes 64 MAC processors, divided into 4 groups to perform 4 different batch operations, with each group of 16 MAC processors being independently distributed. The forward transform data of the WNram 711 needs to be sent to these 64 MAC processors simultaneously, so that they can perform bitwise multiplication and accumulation operations with different Winograd weights. Therefore, the WNram 711 sends the forward transform data by broadcasting or distributing routes. Due to the large output load, in order to ensure driving capability and timing, the forward transform data of the WNram 711 is broadcast or distributed through two levels of N1 and N2 routes. First, it is sent to 4 N1 nodes, each N1 node broadcasts or distributes routes to 4 N2 nodes, and each N2 node then broadcasts or distributes routes to 4 MAC processors.

[0136] ITU 715 reads the bitwise multiplication data from WRram 714 according to the decoded instructions, performs the inverse transformation of the bitwise multiplication data, that is, performs A... T The LA operation is performed to obtain the convolution result, which is temporarily stored in Rram 716.

[0137] Figure 13 A schematic diagram of ITU 715 is shown. ITU 715 includes an input buffer 1301, a register file 1302, an adder group 1303, and an output buffer 1304.

[0138] When ITU 715 receives an instruction to load bitwise multiplication data from WRram 714, input buffer 1301 acts as a first-in-first-out queue buffer to temporarily store bitwise multiplication data based on the input bandwidth. The bitwise multiplication data loading phase continues until all data reception is complete. Convolutional filters of different sizes are configured with fixed and independent buffer resource allocation and input counting. The entire process is controlled by instructions sent by IDU 708.

[0139] According to the decoded instructions, register file 1302 retrieves the temporarily stored bitwise multiplication data from input buffer 1301 in a fixed order of operations and stores it at a specific address in register file 1302. These bitwise multiplication data stored at a specific address in register file 1302 become addition operands. Similarly, to solve the resource dependency problem, register file 1302 has two storage units of the same size, ping storage unit 1305 and ping storage unit 1306. The i-th addition operand and the resulting convolution are temporarily stored in ping storage unit 1305, the (i+1)-th addition operand and the (i+1)-th convolution result are temporarily stored in ping storage unit 1306, and the (i+5)-th addition operand and the (i+5)-th convolution result are temporarily stored in ping storage unit 1305, overwriting the i-th addition operand and the i-th convolution result. Register file 1302 stores data according to this rule.

[0140] Adder group 1303 reads addition operands sequentially from specific addresses in register file 1302 according to the decoded instructions and performs addition operations. Similar to adder group 1003, adder group 1303 consists of two groups to correspond to the addition operation scheduling direction. Each group includes 16 adders to correspond to the vectorization direction. Each adder is an FP32 adder, performing addition operations in the inverse transform of Winograd convolution in a specific order along the channel direction of the bitwise multiplication data. This specific order is to first calculate the left multiplication matrix A of the Winograd convolution. T The addition is performed, followed by the addition of the right multiplication matrix A of the Winograd convolution, and finally the convolution result is generated and stored back into register file 1302. The order of operations, register allocation, and operation time are all related to the size of the convolution filter and are controlled by instructions sent by IDU 708. This operation stage is data-dependent with the aforementioned stage of loading bitwise multiplication data and is executed in a pipelined manner, implemented by hardware counting.

[0141] Output buffer 1304 is also a first-in-first-out queue buffer, used to temporarily store the convolution results sequentially from ping storage unit 1305 and ping storage unit 1306. This output stage depends on the overall completion of the computation stage in order to output the corresponding buffered results based on the output bandwidth.

[0142] Besides Winograd convolution, computing device 601 can also perform all neural network-related operations. ALU 717 performs two main tasks based on the decoded instructions: the first is convolution fusion operations, which can be completed on-chip with the convolutional layers without relying on additional data. These operations include activation, biasing, orientation, and accumulation. The second task is non-convolutional operations. The results of the ALU 717 are also temporarily stored in Rram 716. The presence of ALU 717 ensures that all operations in the convolutional neural network can be fully implemented in computing device 601, giving computing device 601 the versatility and completeness of neural networks.

[0143] The RDMA 706, based on the decoded instructions, retrieves the convolution result from the RRAM 716 and outputs it to the DRAM 604, thus completing the entire convolution operation. Similarly, the RDMA 706 can also retrieve other operation results generated by the ALU 717 from the RRAM 716 and output them to the DRAM 604, based on the decoded instructions. In this embodiment, the output bandwidth of the RRAM 716 is w bytes, and it also includes four memory arrays. Each memory array includes 4×d ​​4×w bit memory cells, i.e., a width of 512 bits and a depth of 4096. Therefore, the size of each memory array is 256KB, and the size of the RRAM 716 is 1MB. Each memory array has an input / output dual-port bandwidth of 64B, and the address space in the depth direction is divided into 16 parts, each with an address space of 256, used to store the results of neuron multiplexing directions.

[0144] Figure 14 The diagram illustrates the connection relationships of the RAM 716. The input ports of the RAM 716 are connected to the ITU 715 and ALU 717, receiving their output data. Since convolution and other operations are not performed, these two input ports do not need to operate simultaneously. Therefore, the input bandwidth of each memory array is maintained at 64 bytes, and this 64-byte bandwidth is time-division multiplexed to access data from the ITU 715 and ALU 717. The RAM 716 also has two output ports, one connected to the RDMA 706 and the other to the ALU 717. After the ALU 717 completes its calculation, the RAM 716 sends the calculation result to the DRAM 604 via the RDMA 706. Therefore, time-division multiplexing is also used at the output end, utilizing the 64-byte output bandwidth to achieve data transmission to the RDMA 706 and ALU 717.

[0145] This invention utilizes the characteristics of the Winograd algorithm in its hardware design to achieve accelerated versatility. It proposes a pipelined operation method to speed up Winograd convolution operations and fully leverages reusable resources through time-division multiplexing and broadcast routing during hardware implementation. The proposed hardware structure is compatible with the Winograd convolution algorithm, offering technical advantages such as guaranteed network accuracy, performance acceleration, area reduction, and power consumption reduction.

[0146] Depending on the application scenario, the electronic devices or apparatus of the present invention may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus of the present invention can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present invention can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0147] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this invention is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also refer to the relevant descriptions of other embodiments.

[0148] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0149] In this invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same position or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this invention. Additionally, in some scenarios, multiple units in the embodiments of this invention may be integrated into one unit or each unit may exist physically independently.

[0150] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0151] The foregoing can be better understood in accordance with the following terms:

[0152] Clause A1, a cache for temporarily storing neuron data in a Winograd convolution, the cache comprising four storage arrays, each storage array comprising four storage blocks, each storage block being a storage unit of size d w bits, where d is the number of addresses in the storage unit.

[0153] Clause A2, the cache described in Clause A1, is connected to Neuron Direct Memory Access (NDMA) for loading neuron data from off-chip memory.

[0154] Clause A3: A cache as described in Clause A1 is connected to a forward transform unit, which reads a data block from the neuron data from the cache each time to perform an addition operation in the forward transform of the Winograd convolution, wherein the size of the data block is [l 4 4], where l is the vectorization length.

[0155] Clause A4, the cache as described in Clause A3, wherein the size of the neuron data is [C i H i W i The storage capacity of the cache is Where α is Directional coefficient, β is H i W i Directional coefficient.

[0156] Clause A5, caching as described in Clause A4, where α is 4 and β is 64.

[0157] Clause A6, caching as described in Clause A3, wherein l is between 16 and 64.

[0158] Clause A7, caching as described in Clause A4, where l is 16.

[0159] Clause A8, caching as described in Clause A1, where w is 128.

[0160] Clause A9, caching as described in Clause A1, where d is 1024.

[0161] Clause A10, a computing device including a cache as described in any one of Clauses A1 to A9.

[0162] Clause A11, an integrated circuit device including a computing device as described in Clause A10.

[0163] Clause A12, a board including an integrated circuit device as described in Clause A12.

[0164] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A cache for temporarily storing neuron data in a Winograd convolution, the cache comprising four storage arrays, each storage array comprising four storage blocks, each storage block having a size of d w-bit storage units, where d is the number of addresses in the storage units, the cache being connected to a forward transform unit for providing the neuron data to the forward transform unit to support the forward transform unit in performing a forward transform operation of the Winograd convolution, wherein the forward transform unit reads a data block from the cache each time to perform an addition operation in the forward transform of the Winograd convolution, wherein the size of the data block is... , The vectorized length is [C]. i H i W i The storage capacity of the cache is ,in for directional coefficient, For H i W i Directional coefficient.

2. The cache according to claim 1, connected to neuron direct memory access, the neuron direct memory access being used to load the neuron data from off-chip memory.

3. The cache according to claim 1, wherein It is 4. It is 64.

4. The cache according to claim 1, wherein Between 16 and 64.

5. The cache according to claim 1, wherein It is 16.

6. The cache according to claim 1, wherein w is 128.

7. The cache according to claim 1, wherein d is 1024.

8. A computing device comprising a cache according to any one of claims 1 to 7.

9. An integrated circuit device comprising the computing device according to claim 8.

10. A board comprising the integrated circuit device according to claim 9.

Images