An architecture supporting arbitrary size and shape convolution kernel computation

Abstract

The application provides an architecture supporting calculation of convolution kernels of any size and shape, comprising an input data preparation part for obtaining, pre-storing and splicing input pixel data, and obtaining weight data and layer parameters, the weight data possibly involving offline segmentation or splitting of the convolution kernel into several sub-kernels; a zigzag data flow data preparation part for flexible arrangement of the pixel data obtained by the input data preparation part, wherein a horizontal data movement module and a vertical data movement module are decoupled from each other, so that the architecture has flexibility in supporting calculation of convolution kernels of any size and shape, without causing redundant calculation and data transmission; a calculation part for convolution of target pixel data and target weight data in a multiply-accumulate array according to the layer parameters; an output data rearrangement and storage part for arrangement and storage of output pixels; and a control part for controlling all the above-mentioned parts to realize layer-by-layer calculation, especially control of calculation of different sub-kernels.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an architecture that supports convolution kernel computation of arbitrary size and shape. Background Technology

[0002] Convolutional neural networks (CNNs) are widely used in various computer vision tasks. In recent years, inspired by the multi-head self-attention layer, a key operation in visual deformers, CNNs have introduced large kernel convolutions to expand their receptive domain, enabling them to capture global patterns in input data and achieve higher performance. For example, RepLKNet, one of the pioneers, used a 31×31 convolutional kernel to demonstrate the effectiveness of large kernel convolutions in improving detection accuracy, even outperforming SwinTransformer on some downstream tasks. Due to the high computational complexity of large convolutional kernels, some network architectures have been proposed using asymmetric convolutions to reduce computation, such as Inception v2, v3, v4, EACNet, and ACR2UNet. Furthermore, Neural Architecture Search (NAS) techniques are used to find optimal neural network architectures for various deep learning tasks (such as image segmentation), and large kernel convolutions and asymmetric convolutions frequently appear in the search results. Large convolutional kernels are also widely applicable to certain application scenarios, such as electrocardiogram detection, biomedical segmentation, and remote sensing. Asymmetric convolutional kernels are also widely used in fields such as audio and text processing. However, directly using hardware accelerators that support small-kernel (3×3) neural networks to support large-kernel convolutions or asymmetric convolutions can introduce sparsity, leading to low computational efficiency. Furthermore, the repetitive data required for subsequent computations due to window sliding is typically buffered in on-chip memory, which incurs significant storage overhead as the convolution kernel size increases. Therefore, a hardware architecture capable of efficiently supporting convolution computations with multi-scale, arbitrarily shaped kernels and reducing on-chip storage overhead is urgently needed.

[0003] To efficiently support convolutional kernels of arbitrary size and shape, existing techniques can be categorized as follows:

[0004] (1) Hardware structure for splitting and padding large convolution kernels. This structure splits the large convolution kernel into several 3×3 sub-kernels, and then uses an accelerator specially designed for 3×3 convolution to directly support it. Since the size of the convolution kernel may not be able to be completely split into several 3×3 sub-kernels, the split asymmetric kernels need to be padded with zero elements to ensure workload balance, which leads to unacceptable computational inefficiency.

[0005] (2) Designing dedicated hardware structures to support large kernels or asymmetric kernels of specific sizes. This is achieved by splitting large convolutional kernels into several sub-kernels of a fixed size (the fixed size is determined based on the network requirements; the sub-kernels can be 1×k or k×1 asymmetric convolutional kernels, or k×k convolutional kernels of a specific size, where k is usually the common divisor of all large kernel sizes to be supported), and then using basic units to compute the sub-kernels. Further hardware connections between these basic units enable support for large kernels. However, due to their fixed hardware architecture, these designs cannot support a wide range of convolutional kernel sizes, and the hardware complexity increases with the number of supported convolutional kernel sizes.

[0006] (3) Hardware architecture that uses row buffers for overlapping data storage. When this type of hardware architecture extends support for large kernel convolutions, it will result in a considerable resource overhead because the buffer size is proportional to the kernel size.

[0007] (4) It can expand the kernel size of the systolic array. Because in the past, it mainly supported 3×3 and 5×5 kernel sizes, which made the corresponding systolic array design relatively fixed and difficult to expand, or the additional hardware overhead of expansion was very large.

[0008] In addition to the shortcomings of the technical solutions mentioned above, existing convolution kernel segmentation architectures, such as directly decomposing two-dimensional large kernel convolutions into one-dimensional convolutions, also lead to a significant decrease in accuracy. Summary of the Invention

[0009] This invention provides an architecture that supports convolution kernel computation of arbitrary size and shape, in order to solve the problem that existing technologies cannot efficiently support convolution kernel computation of arbitrary size and shape.

[0010] The architecture includes:

[0011] The input data preparation section is used to acquire a pixel data block of the input feature map and read the input pixel data required by the zigzag data stream in each cycle. For pixel data that cannot be read in one cycle, it is pre-stored and concatenated, and finally outputs the prepared pixel data; it acquires the weight data of the kernel mapping and finally outputs the target weight data; it acquires the layer parameter data to configure the parameter information required in the convolutional layer calculation process, such as the input feature map size, convolutional kernel size, stride, and the starting address of the input pixel in the input buffer.

[0012] The Z-shaped data stream preparation section is used to flexibly arrange the pixel data acquired by the input data preparation section into a Z-shaped data stream. The Z-shaped data stream arrangement includes performing horizontal data movement processing, vertical data movement processing, or direct transmission on the prepared pixel data, and outputting the target pixel data.

[0013] The calculation section is used to perform convolution calculation on the target pixel data and the target weight data in a multiply-accumulate array according to the layer parameters to obtain the output pixel data;

[0014] The output data rearrangement and storage section is used to rearrange and store the output pixel data generated by the calculation section according to the data reading requirements of the next layer calculation, and then send it back to the off-chip.

[0015] The control section is used to control the input data preparation section, the zigzag data stream preparation section, the calculation section, and the output data rearrangement and storage section.

[0016] Preferably, the input data preparation section includes:

[0017] An input buffer is used to read pixel data blocks of the input feature map from off-chip and store them on-chip. The size of the pixel data block is determined by a cyclic tiling variable, which represents the size of the transmitted and buffered data.

[0018] An input pixel reading module is used to generate the address of the input pixel data required for each cycle of the zigzag data stream in the input buffer, and read the corresponding input pixel data from the input buffer according to the address.

[0019] Preferably, the input data preparation section further includes:

[0020] A pre-fetch register is used to store input pixel data read from the input buffer. When the input pixel data meets the data requirements of a single cycle of the zigzag data stream, it is directly output as pre-fetch pixel data. Otherwise, some pixel data is pre-fetched, and in the cycle that requires the corresponding data, it is concatenated with other required data at the new address and sent together as pre-fetch pixel data to the pre-fetch register to meet the data requirements of a single cycle of the zigzag data stream.

[0021] Preferably, the input data preparation section further includes:

[0022] A convolutional layer parameter buffer, which is used to obtain the convolutional layer parameters corresponding to the input pixel data;

[0023] A weight buffer is used to store the weight data corresponding to the input pixel data and send the weight data to the computation part. The off-chip storage order of the weight data is determined by the computation requirements of the zigzag data stream. Different offline processing is performed according to the stride and kernel size in the convolutional layer parameters. When the stride is 1, the elements in the convolutional kernel are directly stored in a zigzag arrangement. Otherwise, the convolutional kernel is split according to the mapping relationship between the input pixels and the weight data, and then each sub-kernel is arranged in a zigzag arrangement. The arrangement order between the sub-kernels is derived from the zigzag data stream to meet the data requirements of each cycle. When the convolutional kernel size is less than 2×Pox, the elements in the convolutional kernel are directly stored in a zigzag arrangement. Otherwise, the convolutional kernel is divided according to the computational parallelism, and then each sub-kernel is arranged in a zigzag arrangement. Pox represents the computational parallelism in the output feature map row direction. The arranged weight data is stored in off-chip storage in an order that meets the computational requirements, waiting to be read from the on-chip weight buffer.

[0024] Preferably, the zigzag data stream preparation section includes a first multiplexer, which is configured to:

[0025] According to the data reuse rules of the Z-shaped data stream, the prepared pixel data, after passing through the horizontal data movement module, the vertical data movement module, or direct transmission operation, is transmitted to the calculation part. The data reuse rules of the Z-shaped data stream include horizontal data reuse rules and vertical data reuse rules. The horizontal data reuse rule refers to the fact that, due to the parallelism in the row and column dimensions of the input feature map, the weight data participates in the calculation according to the Z-shaped data stream rules in time order. When calculating in the horizontal order, the input pixels of multiple parallel processing units will overlap in the horizontal dimension, and the overlapping data can be reused through horizontal movement. The vertical data reuse rule refers to the fact that, when calculating in the vertical order, the input pixels of multiple parallel processing units will overlap in the vertical dimension, and the overlapping data can be reused through vertical movement.

[0026] When the weight data required for the next cycle calculation is in the same row as the current weight data in the convolution kernel, the prepared pixel data passed through the horizontal data movement module is transmitted to the corresponding port of the calculation section according to the horizontal data reuse rule of the convolution window. When the weight data required for the next cycle calculation is in a different row from the current weight data in the convolution kernel, the prepared pixel data passed through the vertical data movement module is transmitted to the corresponding port of the calculation section according to the vertical data reuse rule of the convolution window in the zigzag data stream. The above two operations are accompanied by the need to read new data. This is achieved by directly transmitting the prepared pixel data from the input data preparation section without processing. The read new data and the data passed through the horizontal data movement module or the vertical data movement module are spliced ​​together to form all the input data required by the calculation section of a single cycle.

[0027] Preferably, the Z-shaped data stream preparation section further includes:

[0028] A horizontal data movement module is used to obtain preliminary pixel data from the input data preparation section, determine whether the preliminary pixel data satisfies the horizontal data reuse rule of the Z-shaped data flow, and if so, perform horizontal data movement processing on the preliminary pixel data.

[0029] The vertical data movement module is used to obtain preliminary pixel data from the input data preparation section, determine whether the preliminary pixel data satisfies the vertical data reuse rule of the Z-shaped data stream, and if so, perform vertical data movement processing on the preliminary pixel data.

[0030] Preferably, the calculation part includes:

[0031] The calculation module includes a built-in multiply-accumulate array. The size of the multiply-accumulate array is determined by the loop unrolling variable, which in this invention is Pox×Poy×Pof. Here, Pox, Poy, and Pof represent the computational parallelism in the row, column, and output channel directions of the output feature map, respectively. The calculation module is used to perform convolution calculations on the target pixel data and the target weight data in the multiply-accumulate array according to the convolutional layer parameters to obtain the output pixel data.

[0032] Preferably, the output data rearrangement and storage section includes:

[0033] An output data arrangement module is used to rearrange the output pixel data according to the data reading requirements of the next layer calculation.

[0034] An output buffer is used to store the output pixel data after the output data arrangement module rearranges the data.

[0035] Preferably, the control section includes:

[0036] A layer-by-layer control module is used to control the input data preparation part, the zigzag data stream preparation part, the calculation part, and the output data rearrangement and storage part to realize layer-by-layer convolution calculation.

[0037] The layer-by-layer control module includes a sub-core control module. The sub-core control module controls the reading order of input pixels and weight data to ensure that the data flow of each sub-core still meets the Z-shaped data flow pattern under large convolution kernel size and stride, thereby reusing the existing hardware design.

[0038] Preferably, the specific implementation of the Z-shaped data stream preparation section includes:

[0039] Send the prepared pixel data to the corresponding register;

[0040] Based on the input pixel reuse pattern of the horizontal movement of the convolution window during convolution calculation, the designed Z-shaped data flow provides opportunities for horizontal data reuse. Specifically, the prepared pixel data parallel to the row direction of the input feature map is assigned to the adjacent register array on the left or right in the next cycle to achieve data reuse. The direction of assignment is determined by the parity of the number of rows of the corresponding weight data in the convolution kernel. Odd-numbered rows go to the left and even-numbered rows go to the right.

[0041] By introducing computation at the kernel mapping inflection point, the opportunity for vertical data reuse is increased. The prepared pixel data, which runs parallel along the column direction of the input feature map, is assigned to the adjacent register array on the left in the next cycle to achieve reuse.

[0042] This invention provides an architecture supporting convolution kernel computation of arbitrary size and shape, including an input data preparation section for acquiring, pre-storing, and concatenating input pixel data, and for acquiring weight data and layer parameters. The weight data may involve offline segmentation or splitting of the convolution kernel into several sub-kernels so that the data flow of each sub-kernel satisfies the Z-shaped data flow pattern, reusing existing hardware resources. A Z-shaped data flow preparation section is used to flexibly arrange the pixel data acquired by the input data preparation section, wherein the horizontal data movement module and the vertical data movement module are decoupled, giving the architecture the flexibility to support convolution kernel computation of arbitrary size and shape without generating redundant computation and data transmission. A computation section is used to convolve the target pixel data and target weight data in a multiply-accumulate array according to the layer parameters. An output data rearrangement and storage section is used to arrange and store the output pixels. A control section is used to control all the above sections to achieve layer-by-layer computation, especially the control of different sub-kernel computations. Attached Figure Description

[0043] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of an architecture proposed in this invention that supports convolution kernel computation of arbitrary size and shape;

[0045] Figure 2 This is a schematic diagram of a multiply-accumulate array;

[0046] Figure 3 This is a schematic diagram of a zigzag data stream with a step size of 1.

[0047] Figure 4 This is a schematic diagram of a data scheduling instance with a step size of 1.

[0048] Figure 5 This is a schematic diagram of the convolution kernel segmentation strategy when the kernel mapping is large;

[0049] Figure 6 This is a schematic diagram of a zigzag data stream with a step size of 2.

[0050] Figure 7 This is a schematic diagram of the convolution loop after adding a sub-kernel loop with a stride of 2;

[0051] Figure 8 This is a schematic diagram of a data scheduling instance with a step size of 2;

[0052] Figure 9 This is a schematic diagram illustrating the sequential variant of kernel mapping element calculation in this invention;

[0053] Figure 10 This is a diagram illustrating the convolution loop dimension and design variables. Detailed Implementation

[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Figure 10 This is a diagram illustrating the convolution loop dimension and design variables.

[0056] For the main operations of convolution, see [link to main operations]. Figure 10This involves accumulating the weights and the product of the input pixels along different dimensions of the kernel and feature map. The descriptive dimensions discussed below can be categorized into three types. First, the size of the feature and kernel map is represented by the convolutional loop dimension (N*). Second, the size of the buffered data is determined by the loop tiling variable (T*), which also affects the latency of DRAM access. In the design of this invention, all input channels of an entire row of input pixels and feature maps are buffered, i.e., Tix = Nix, Tif = Nif (Tix represents the block size along the row direction of the input feature map, Nix represents the row size of the input feature map; Tif represents the block size along the channel direction of the input feature map, Nif represents the channel size of the input feature map), to generate a complete row of results. Third, parallel computation is guided by the loop unrolling variable (P*), where Pox, Poy, and Pof represent the number of parallel computations along the row, column, and channel directions of the output feature map, respectively, and are set to 8, 8, and 16 in this embodiment.

[0057] Inspired by the rapid development of Vision Transformers (ViTs), Convolutional Neural Networks (CNNs) have introduced multi-scale convolutional kernels to expand their receptive domain and learn details at different levels, enabling them to achieve higher performance. Meanwhile, some network architectures introduce asymmetric convolutions (kernel sizes of 1×n or n×1) to reduce computational cost. However, for networks with multi-scale convolutional kernels, the overlapping data required for subsequent computations due to convolution window sliding is typically buffered in on-chip memory. This can lead to significant storage overhead when the kernel size is large. Furthermore, existing work lacks flexible support for asymmetric convolutions, posing a significant challenge to the efficient deployment of neural networks containing asymmetric convolutions. This embodiment addresses the problem of designing a hardware architecture friendly to convolutional kernels of arbitrary size and shape, reducing the need for large amounts of overlapping data storage or data transmission, especially when the kernel size is large, thereby saving on-chip storage overhead or reducing data transmission latency, while efficiently implementing networks containing large and asymmetric kernels on a Field Programmable Gate Array (FPGA) platform.

[0058] To address the issue that existing methods, which primarily involve dividing a large kernel into smaller kernels (3×3) to support large-kernel convolution calculations, may introduce sparsity and thus reduce computational efficiency, this embodiment designs a hardware architecture that supports convolution kernel calculations of arbitrary size and shape. Specifically, in this architecture, the loop unrolling variables are not parallelized within the kernel mapping window, thus hardware utilization is not affected by the convolution kernel size. Furthermore, since increasing the convolution kernel size will lead to a large amount of on-chip storage or data transfer requirements for overlapping data, this invention designs a novel Z-shaped data flow. This introduces convolution kernel inflection point calculations only by changing the element calculation order of the convolution kernel window, increasing the opportunity for vertical data reuse. Pixels required for different weight data calculations can directly reuse input pixels from adjacent registers in the previous cycle through horizontal or vertical data movement, thereby reducing hardware resource overhead for storing overlapping data or data transfer latency, especially when the convolution kernel size is large. Simultaneously, because horizontal and vertical data movements are decoupled, it can efficiently support convolution kernels of arbitrary size and shape without introducing additional computation and data transfer. For cases where the stride is not 1, the convolution kernel is split into different sub-kernels. The data flow corresponding to the sub-kernels still follows the zigzag data flow pattern, allowing for the reuse of existing hardware designs. By controlling the computation of the sub-kernels and accumulating the computation results of different sub-kernels, the final output pixel data can be obtained.

[0059] Figure 1 This is a schematic diagram of an architecture proposed in this invention that supports convolution kernel calculations of arbitrary size and shape.

[0060] The architecture is mainly divided into five parts: control part 500, input data preparation part 100, zigzag data stream preparation part 200, calculation part 300, and output data rearrangement and storage part 400.

[0061] The control unit 500 is responsible for all the control required for the layer-by-layer computation of the convolutional neural network. In particular, the control unit 500 includes a layer-by-layer control module 510, which contains a sub-kernel control module 511. This module controls the reading order of input pixels and weight data to ensure that the data flow of each sub-kernel still meets the Z-shaped data flow pattern under the condition of large convolutional kernel size and stride. At the same time, it can reuse existing hardware design and save resource consumption.

[0062] The input data preparation section 100 includes an input buffer 110, an input pixel reading module 120, a preparatory register 130, a convolutional layer parameter buffer 140, and a weight buffer 150. The input pixel reading module 120 generates addresses corresponding to the data required for each cycle of the zigzag data stream and reads the data at the corresponding addresses from the input buffer 110. This part of the data needs to determine whether it can provide all the input pixel data required for each cycle of the zigzag data stream. Pixel data that cannot be read within one cycle is pre-stored and concatenated, and finally, preparatory pixel data is output. This process is performed in the preparatory register 130. The convolutional layer parameters required during the calculation process are obtained from the convolutional layer parameter buffer 140. The weight data required for the calculation is obtained from the weight buffer 150. Depending on the stride and kernel size in the convolutional layer parameters, different offline processing methods are used for the convolutional kernel. When the stride is 1, the elements in the convolution kernel are directly stored in a Z-shaped arrangement. Otherwise, the convolution kernel needs to be split according to the mapping relationship between the input pixels and the weight data, and then each of the split sub-kernels is arranged in a Z-shape. The arrangement order between the sub-kernels is derived from the Z-shaped data stream to meet the data requirements of each cycle. When the convolution kernel size is less than 2×Pox, the elements in the convolution kernel are directly stored in a Z-shaped arrangement. Otherwise, the convolution kernel needs to be divided according to the computational parallelism. The size of the first sub-kernel obtained by the division is an integer multiple of the parallelism, and then each of the divided sub-kernels is arranged in a Z-shape. The arranged weight data is stored in off-chip storage in an order that meets the computational requirements, waiting to be read by the on-chip weight buffer 150. The weight data arrangement process is completed offline, and then the weight data of different sub-kernels is read sequentially to the computation part 300 according to the weight requirements of the data stream.

[0063] The Z-shaped data stream preparation section 200 includes a horizontal data movement module 210 and a vertical data movement module 220. According to the data multiplexing rules of the Z-shaped data stream, the prepared pixel data, after passing through the horizontal data movement module 210, the vertical data movement module 220, or direct transmission operations, is transmitted to the calculation section 300.

[0064] The data reuse rules of the Z-shaped data stream include horizontal data reuse rules and vertical data reuse rules. The horizontal data reuse rule refers to the fact that, due to the parallel processing of the input feature map in both row and column dimensions, the weight data participates in the calculation according to the Z-shaped data stream in time order. When calculating in the horizontal order, the input pixels of multiple parallel processing units will overlap in the horizontal dimension, and the overlapping data can be reused by horizontal movement. The vertical data reuse rule refers to the fact that, when calculating in the vertical order, the input pixels of multiple parallel processing units will overlap in the vertical dimension, and the overlapping data can be reused by vertical movement.

[0065] When the weight data required for the next cycle calculation is in the same row as the current weight data in the convolution kernel, i.e., the required weight data is not located at the inflection point of the convolution kernel, the prepared pixel data passed through the horizontal data movement module 210 is transmitted to the corresponding port of the calculation section 300 according to the horizontal data reuse rule of the convolution window. When the weight data required for the next cycle calculation is in a different row from the current weight data in the convolution kernel, i.e., the required weight data is located at the inflection point of the convolution kernel, the prepared pixel data passed through the vertical data movement module 220 is transmitted to the corresponding port of the calculation section 300 according to the vertical data reuse rule of the convolution window in the zigzag data stream. The above two operations are accompanied by the need to read new data, which is achieved by directly transmitting the prepared pixel data from the input data preparation section 100 without processing. The read new data and the data passed through the horizontal data movement module 210 or the vertical data movement module 220 are spliced ​​together to form all the input data required by the calculation section 300 in a single cycle.

[0066] The horizontal data movement module 210 is used to obtain preliminary pixel data from the input data preparation part 100, determine whether the preliminary pixel data satisfies the horizontal data reuse rule of the Z-shaped data flow, and if so, perform horizontal data movement processing on the preliminary pixel data.

[0067] The vertical data movement module 220 is used to obtain preliminary pixel data from the input data preparation part 100, determine whether the preliminary pixel data satisfies the vertical data reuse rule of the Z-shaped data flow, and if so, perform vertical data movement processing on the preliminary pixel data.

[0068] The computation section 300 includes a computation module 310. The computation module 310 contains a multiply-accumulate array of size Pox×Poy×Pof; where Pox, Poy, and Pof represent the computational parallelism in the rows, columns, and output channels of the output feature map, respectively. The computation module is used to perform convolution calculations on the target pixel data and the target weight data in the multiply-accumulate array according to the convolutional layer parameters to obtain the output pixel data.

[0069] The output data rearrangement and storage section 400 includes an output data arrangement module 410 and an output buffer 420. The convolution calculation results are arranged in the output data arrangement module 410 to meet the data reading requirements of the next layer calculation, then stored in the output buffer 420, and finally written off-chip.

[0070] Example 1: Z-shaped hardware data flow design with a step size of 1.

[0071] Figure 3This is a schematic diagram of a Z-shaped data stream with a step size of 1.

[0072] It should be noted that, Figure 3 Taking a 3×3 convolution kernel as an example, PE represents a processing element. In the hardware architecture of this invention, a multiply-accumulate unit is used as the processing unit, which is computed in parallel within an input feature map (Pox, Poy) and multiple kernel maps (Pof).

[0073] Figure 3 The detailed operation is demonstrated; since parallel computation is not performed within the kernel mapping, the multiplication of weights and corresponding input data is carried out in a given time order. The specific data arrangement can be divided into two parts, described below:

[0074] Horizontal data reuse: using PE 0i The calculation data stream for the first three cycles (0i = 00, 01, 02, 03) will be described in detail using this example. First, the four input pixels are loaded from the input buffer into the register and then sent to the corresponding PE. 0i These pixels are reused in the next cycle by the register array to their left, while the rightmost register in each array begins reading input pixels from the buffer, such as... Figure 3 The black fill indicates this. This operation is repeated in subsequent cycles until the calculation of one row of data in the kernel map is complete. For a 3×3 kernel in the diagram, this part of the calculation is completed in the third cycle. When calculating even-numbered rows in the kernel map, the input pixels perform the same operation as described above, i.e., they are reused from the right-hand adjacent register array, and the new input pixel is read into the leftmost register array.

[0075] Vertical data reuse: The Z-shaped data flow proposed in this invention provides new data reuse opportunities by introducing the calculation of inflection points in kernel mapping, such as... Figure 3 The black box and bold black arrow in the right-hand image indicate this. Specifically, in the fourth cycle, the previous cycle sent to the PE... 1i The pixels are sequentially reused by the adjacent register array to the left, that is, sent to the corresponding PE. 0i Similarly, PE 2i The pixels are sent to PE 1i PE 3i The pixels are sent to PE 2i At this point, four new pixels are input from the input buffer into the rightmost register array, as shown by the black fill in the figure.

[0076] Figure 2 This is a schematic diagram of a multiply-accumulate array.

[0077] It should be noted that, Figure 2In this context, MAC stands for Multiply-and-accumulate units.

[0078] The design of the multiply-accumulate array is guided by the proposed zigzag data stream and cyclic expansion variables. It consists of Pox × Poy × Pof multiply-accumulate units, each consisting of a multiplier and an accumulator. Because all input channels are fully buffered in the architecture of this invention, partial sums are consumed within the multiply-accumulate units until the final result is obtained. Data in the input buffer is shared across Pof multiply-accumulate units, and data in the weight buffer is shared across Pox × Poy multiply-accumulate units. The computation result is sent to the output buffer via an output data rearrangement module.

[0079] Figure 4 This is a schematic diagram of a data scheduling instance with a step size of 1.

[0080] It should be noted that, Figure 4 Using a 3×3 convolution kernel as an example, we will demonstrate the specific hardware data flow scheduling.

[0081] Figure 4 The top section displays the pixel data at different locations in the input feature map (named R0C, where R represents row and C represents column, e.g., 102 represents the pixel data in row 1 and column 2) and the weights (K1, K2, etc.) at different locations in the kernel mapping; the middle section shows the specific data flow arrangement in different periods of the PE, where different Ki (here i = 1-9, i is a positive integer) are assigned to the PE in each period. 00 -PE 33 The corresponding input pixels in the PE are multiplexed. The input pixels in the PE flow over time according to a zigzag data stream pattern, and the new data to be read in each cycle is... Figure 4 The text in the middle is bolded; the bottom is the input buffer, the size of which is determined by the loop unrolling variable; the left side is the data address, and each address reads data of Pox × Poy, where Pox is the width of each buffer and Poy is the number of buffers.

[0082] In the first cycle, data at address 0 is read from the input buffer and sent to the computation unit via a register. Figure 4 China PE 00 -PE 33 );

[0083] In the second cycle, data at address 1 is read from the input buffer and sent to the computation unit via a register. Simultaneously, the data at that address is stored in reserve register 1.

[0084] In the third cycle, data is read from the preparatory register 1 and sent to the computation unit via the register. Simultaneously, data at address 2 is read from the input buffer and stored in preparatory register 2.

[0085] In the fourth cycle, address 3 is read from the input buffer (to obtain 505 and 506), and concatenated with the data in the preparatory register 2 (pre-stored 503 and 504), and sent to the calculation unit via the register;

[0086] In the fifth cycle, address 0 is read from the input buffer (to obtain 202, 302, and 402), and concatenated with the data in the preparatory register 2 (the pre-stored 502), and then sent to the calculation unit via the register;

[0087] In the sixth cycle, address 0 is read from the input buffer (to obtain 201, 301, and 401), and concatenated with the data in the preparatory register 2 (the pre-stored 501), and then sent to the calculation unit via the register;

[0088] In the seventh cycle, data at address 2 is read from the input buffer and sent to the computation unit via a register;

[0089] In the eighth cycle, address 3 is read from the input buffer (to obtain 505 and 605), and concatenated with the data in the preparatory register 1 (305 and 405 stored in advance), and sent to the calculation unit via the register;

[0090] In the ninth cycle, address 3 is read from the input buffer (to obtain 506 and 606), and concatenated with the data in the preparatory register 1 (306 and 406 stored in advance), and then sent to the calculation unit via the register;

[0091] Note that after the splicing operation in the above cycle, data will be selected according to the calculation needs. Therefore, even if the eighth and ninth cycles seem to be performing the same operation, the data sent to the calculation unit is different.

[0092] As illustrated in the examples above, the small amount of data required for computation may be pre-stored in a reserve register to avoid read conflicts. Thanks to the data reuse opportunities provided by the proposed zigzag data flow, this invention eliminates the need for linear buffers to store overlapping data, thereby significantly reducing storage overhead. Furthermore, due to the decoupling of horizontal and vertical data reuse implementations, this invention offers the flexibility to support computation of convolution kernels of arbitrary sizes and shapes without generating redundant computation and data transmission.

[0093] Figure 5 This is a schematic diagram of the convolution kernel segmentation strategy when the kernel mapping is large.

[0094] It should be noted that, Figure 5 In Nkx_sub1, Nkx_sub1 represents the amount of data stored at each address within each buffer.

[0095] Larger kernel mappings (Nkx > 2 × Pox) need to be partitioned within the architecture of this invention to avoid additional resource consumption. This invention sets the size of the first sub-core to an integer multiple of the loop unrolling variable, for example... Figure 5 In this configuration, Nkx_sub1 = Pox, ensuring that the corresponding input pixels required for calculations by other sub-cores can be accurately obtained from the same address in different parallel buffers without read conflicts. Each sub-core after segmentation can flow data independently according to a zigzag data flow pattern.

[0096] Example 2: Z-shaped hardware data flow design with a step size of 2.

[0097] Figure 6 This is a schematic diagram of a Z-shaped data stream with a step size of 2.

[0098] It should be noted that, Figure 6 Taking a 3×3 convolution kernel as an example, PE in the figure represents a processing element; Nkg represents the number of sub-kernels after the convolution kernel is split.

[0099] For the step size of 2, it is necessary to follow the procedure as follows: Figure 6 The convolution kernel is split into sub-kernel maps as shown in the diagram. The computation order of elements in each sub-kernel map still follows a zigzag data flow (if the size of the split convolution kernel is small, it may not be a complete zigzag). Due to the flexibility of the architecture, the asymmetric convolution kernels generated during the splitting process can be efficiently supported without increasing redundant computation. At this time, each sub-kernel can flow data independently according to the pattern when the stride is 1, thus reusing the hardware logic when the stride is 1.

[0100] Figure 7 This is a schematic diagram of the convolution loop after adding a sub-kernel loop with a stride of 2.

[0101] It should be noted that, Figure 7 In this context, Nkx and Nky represent the sub-kernel window size, Nkg represents the number of sub-kernels, Nif represents the total number of input channels, Nox and Noy represent the row and column dimensions of the output feature map, and Nof represents the total number of output channels.

[0102] Specifically, the convolution loop unrolling of this invention adds Kg in addition to Kx and Ky to traverse different sub-kernel mappings generated by cutting or splitting. Since it only disrupts the calculation order of elements in the kernel mapping, it does not affect the result. Specifically, the final convolution result is obtained by accumulating the calculation results of different sub-kernels. In addition, the calculation order between different sub-kernels is determined based on whether the amount of data that the buffer can provide simultaneously can meet the calculation requirements. The calculation order in this invention can ensure that all the data required for parallel calculation in each cycle is provided normally.

[0103] Figure 8 This is a schematic diagram of a data scheduling instance with a step size of 2.

[0104] It should be noted that, Figure 8 This example uses a 5×5 convolution kernel to demonstrate the specific hardware data flow scheduling. The 5×5 kernel was chosen so that the calculation order of elements within the sub-kernel can present a complete Z-shape for easier understanding.

[0105] Figure 8 The top section displays the pixels at different locations in the input feature map (named R0C, where R represents row and C represents column; for example, 102 represents the pixel data in row 1, column 2) and their weights (K1, K2, etc.) in the kernel mapping. Since the stride is 2, the convolution kernel needs to be split here; each sub-kernel is shown below the arrow. The middle section shows the specific data flow arrangement in different PE cycles, where different Ki values ​​(here, i = 1-25, where i is a positive integer) are mapped by the PE in each cycle. 00 -PE 33 In the input pixel reuse mechanism, the input pixels in the PE are divided into four parts according to the time flow pattern of each sub-core, and each part still follows the Z-shaped data flow pattern. The new data to be read in each cycle is... Figure 8 The values ​​in bold represent the final calculation result, which is the sum of the calculation results for each sub-core. The bottom layer shows the input buffer, and the left layer shows the addresses. Since the step size is 2, there is only one required input data every pixel. Therefore, each address needs to read 2×Pox×Poy data, where 2×Pox is the width of each buffer, and Poy is the number of buffers.

[0106] In the first cycle, data at address 0 is read from the input buffer and sent to the computation unit via a register. Figure 8 China PE 00 -PE 33 );

[0107] In the second cycle, data at address 1 is read from the input buffer and sent to the computation unit via a register. Simultaneously, the data at that address is stored in reserve register 1.

[0108] In the third cycle, data is read from the preparatory register 1 and sent to the computation unit via the register. Simultaneously, data at address 4 is read from the input buffer and stored in preparatory register 2.

[0109] In the fourth cycle, address 5 is read from the input buffer (to obtain 909 and 911), and concatenated with the data in the preparatory register 2 (905 and 907 stored in advance), and sent to the calculation unit via the register;

[0110] In the fifth cycle, address 0 is read from the input buffer (to obtain 303, 503, and 703), and concatenated with the data in the preparatory register 2 (903 stored in advance), and then sent to the calculation unit via the register;

[0111] In the sixth cycle, address 0 is read from the input buffer (to obtain 301, 501, and 701), and concatenated with the data in the preparatory register 2 (901 stored in advance), and sent to the calculation unit via the register;

[0112] In the seventh cycle, data at address 4 is read from the input buffer and sent to the computation unit via a register;

[0113] In the eighth cycle, address 5 is read from the input buffer (to obtain 909 and 1109), and concatenated with the data in the preparatory register 1 (509 and 709 stored in advance), and sent to the calculation unit via the register;

[0114] In the ninth cycle, address 5 is read from the input buffer (to obtain 911 and 1111), and concatenated with the data in the preparatory register 1 (511 and 711 stored in advance), and sent to the calculation unit via the register;

[0115] In the tenth cycle, data at address 0 is read from the input buffer and sent to the computation unit via a register;

[0116] In the eleventh cycle, the data in the preparation register 1 is read and sent to the computing unit via the register;

[0117] In the twelfth cycle, address 5 is read from the input buffer (in order to obtain 910), and concatenated with the data in the preparatory register 2 (904, 906 and 908 stored in advance), and sent to the calculation unit via the register;

[0118] In the thirteenth cycle, address 0 is read from the input buffer (to obtain 302, 502, and 702), and concatenated with the data in the preparatory register 2 (902 stored in advance), and then sent to the calculation unit via the register;

[0119] In the fourteenth cycle, data at address 4 is read from the input buffer and sent to the computation unit via a register;

[0120] In the fifteenth cycle, address 5 is read from the input buffer (to obtain 910 and 1110), and concatenated with the data in the preparatory register 1 (510 and 710 stored in advance), and sent to the calculation unit via the register;

[0121] In the sixteenth cycle, data at address 2 is read from the input buffer and sent to the computation unit via a register;

[0122] In the seventeenth cycle, data at address 3 is read from the input buffer and sent to the computation unit via a register. Simultaneously, the data at that address is stored in the preparatory register 1.

[0123] In the eighteenth cycle, data is read from the preparatory register 1 and sent to the computation unit via the register. Simultaneously, data at address 6 is read from the input buffer and stored in preparatory register 2.

[0124] In the nineteenth cycle, address 7 is read from the input buffer (to obtain 1009 and 1011), and concatenated with the data in the preparatory register 2 (1005 and 1007 stored in advance), and sent to the calculation unit via the register;

[0125] In the twentieth cycle, address 2 is read from the input buffer (to obtain 403, 603, and 803), and concatenated with the data in the preparatory register 2 (1003 stored in advance), and then sent to the calculation unit via the register;

[0126] In the twenty-first cycle, address 2 is read from the input buffer (to obtain 401, 601, and 801), and concatenated with the data in the preparatory register 2 (1001 stored in advance), and then sent to the calculation unit via the register;

[0127] In the twenty-second cycle, data at address 2 is read from the input buffer and sent to the computing unit via a register;

[0128] In the 23rd cycle, the data in the preparation register 1 is read and sent to the computing unit via the register;

[0129] In the twenty-fourth cycle, address 7 is read from the input buffer (in order to obtain 1010), and concatenated with the data in the preparatory register 2 (1004, 1006 and 1008 stored in advance), and sent to the calculation unit via the register;

[0130] In the twenty-fifth cycle, address 2 is read from the input buffer (to obtain 402, 602, and 802), and concatenated with the data in the preparatory register 2 (1002 stored in advance), and sent to the calculation unit via the register;

[0131] For other stride sizes, the convolution kernel can be split into several sub-kernels for extended support, and each sub-kernel still follows the data flow pattern when the stride is 1.

[0132] Figure 9 This is a schematic diagram of the kernel mapping element calculation order variant in this invention.

[0133] Figure 9 A variant of the kernel mapping computation order in a zigzag data flow is shown. Although it appears different from the computation order of this invention, it actually only involves a rotation operation, and the essence of data reuse remains the same. The operations of the embodiments can be completed following a similar process to this invention without requiring innovative design. Flexible choices can be made among these variants according to different architectural design needs.

[0134] The architecture in this embodiment has the following advantages:

[0135] By performing parallel computation along the Pox, Poy, and Pof dimensions, hardware utilization is unaffected by the kernel window size. Furthermore, a novel zigzag data flow is employed. By altering the computation order of elements within the kernel mapping, vertical data reuse opportunities for input pixels at kernel mapping inflection points are added to the horizontal data reuse framework. This alleviates the on-chip storage or data transfer requirements of rapidly increasing overlapping data as the kernel size grows, reducing storage overhead or transmission latency. Moreover, because the horizontal and vertical data reuse implementations are decoupled in the architecture of this invention, it offers the flexibility to support computation of convolutional kernels of arbitrary shapes and sizes without redundant computation and data transfer. Finally, for larger kernel sizes or strides, the kernels can be segmented or split so that the input pixels corresponding to each sub-kernel still satisfy the zigzag data flow pattern, thereby reusing existing hardware structures and saving additional resource overhead. By controlling the sub-kernel computation and accumulating the computation results of different sub-kernels, the final output pixel data can be obtained.

Claims

1. An architecture that supports convolution kernel computation of arbitrary size and shape, characterized in that, The architecture includes: The input data preparation part (100) is used to obtain a pixel data block of the input feature map, and read the input pixel data required by the zigzag data stream in each cycle. For pixel data that cannot be read in one cycle, it is pre-stored and spliced, and finally outputs the prepared pixel data; obtains the weight data of the kernel mapping, and finally outputs the target weight data; obtains the layer parameter data to configure the parameter information required in the convolutional layer calculation process; Z-shaped data stream preparation section (200), the Z-shaped data stream preparation section (200) is used to flexibly arrange the pixel data obtained by the input data preparation section (100) into a Z-shaped data stream, the Z-shaped data stream arrangement includes performing horizontal data movement processing, vertical data movement processing or direct transmission on the prepared pixel data, and outputting target pixel data; The calculation part (300) is used to perform convolution calculation on the target pixel data and the target weight data in a multiply-accumulate array according to the layer parameters to obtain the output pixel data; The output data rearrangement and storage section (400) is used to rearrange, store and send back the output pixel data generated by the calculation section (300) according to the data reading requirements of the next layer calculation. The control section (500) is used to control the input data preparation section (100), the zigzag data stream preparation section (200), the calculation section (300), and the output data rearrangement and storage section (400).

2. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 1, characterized in that, The input data preparation section (100) includes: An input buffer (110) is used to read pixel data blocks of the input feature map from off-chip and store them on-chip. The amount of data in the pixel data blocks is determined by a cyclic tiling variable, which expresses the size of the transmitted and buffered data. The input pixel reading module (120) is used to generate the address of the input pixel data required for each cycle of the zigzag data stream in the input buffer (110), and read the corresponding input pixel data from the input buffer (110) according to the address.

3. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 2, characterized in that, The input data preparation section (100) also includes: A preparatory register (130) is used to store input pixel data read from the input buffer (110). When the input pixel data meets the data requirements of a single cycle of the zigzag data stream, it is directly output as preparatory pixel data. Otherwise, some pixel data is prefetched, and in the cycle that requires corresponding data, it is concatenated with other required data at the new address and sent together as preparatory pixel data to the preparatory register (130) to meet the data requirements of a single cycle of the zigzag data stream.

4. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 3, characterized in that, The input data preparation section (100) also includes: A convolutional layer parameter buffer (140) is used to obtain the convolutional layer parameters corresponding to the input pixel data; A weight buffer (150) is used to store the weight data corresponding to the input pixel data and send the weight data to the calculation part (300). The storage order of the weight data outside the chip is determined by the calculation requirements of the zigzag data stream. Different offline processing is performed according to the stride and kernel size in the convolutional layer parameters. When the stride is 1, the elements in the convolutional kernel are directly arranged and stored in a zigzag pattern. Otherwise, the convolutional kernel is split according to the mapping relationship between the input pixel and the weight data, and each split weight is used to store the weight data. The sub-kernels are arranged in a Z-shape, and the arrangement order between the sub-kernels is derived from the Z-shaped data flow to meet the data requirements of each cycle. When the convolution kernel size is less than 2×Pox, the elements in the convolution kernel are directly arranged in a Z-shape for storage. Otherwise, the convolution kernel is divided according to the computational parallelism, and each sub-kernel obtained by the division is arranged in a Z-shape. Pox represents the computational parallelism in the output feature map row direction. The arranged weight data is stored in off-chip storage in an order that meets the computational requirements, waiting to be read from the on-chip weight buffer (150).

5. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 1, characterized in that, The zigzag data stream preparation section (200) includes a first multiplexer (230), which is configured to: According to the data reuse rules of the Z-shaped data stream, the prepared pixel data, after passing through the horizontal data movement module (210), the vertical data movement module (220), or direct transmission operation, is transmitted to the calculation part (300). The data reuse rules of the Z-shaped data stream include horizontal data reuse rules and vertical data reuse rules. The horizontal data reuse rules refer to the fact that, since the input feature map is parallel in the row and column dimensions, the weight data participates in the calculation according to the Z-shaped data stream rules in the time order. When calculating in the horizontal order, the input pixels of multiple parallel processing units will overlap in the horizontal dimension, and the overlapping data can be reused by horizontal movement. The vertical data reuse rules refer to the fact that, when calculating in the vertical order, the input pixels of multiple parallel processing units will overlap in the vertical dimension, and the overlapping data can be reused by vertical movement. When the weight data required for the next cycle calculation is in the same row as the current weight data in the convolution kernel, the prepared pixel data passed through the horizontal data moving module (210) is transmitted to the corresponding port of the calculation part (300) according to the horizontal data reuse rule of the convolution window; when the weight data required for the next cycle calculation is in a different row from the current weight data in the convolution kernel, the prepared pixel data passed through the vertical data moving module (220) is transmitted to the corresponding port of the calculation part (300) according to the vertical data reuse rule of the convolution window in the zigzag data stream; the above two operations are accompanied by the need to read new data, which is achieved by directly transmitting the prepared pixel data from the input data preparation part (100) without processing. The read new data and the data passed through the horizontal data moving module (210) or the vertical data moving module (220) are spliced ​​together to form all the input data required by the single cycle calculation part (300).

6. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 5, characterized in that, The Z-shaped data stream preparation section (200) also includes: A horizontal data movement module (210) is used to obtain preliminary pixel data from the input data preparation part (100), determine whether the preliminary pixel data satisfies the horizontal data reuse rule of the Z-shaped data flow, and if so, perform horizontal data movement processing on the preliminary pixel data. The longitudinal data movement module (220) is used to obtain preliminary pixel data from the input data preparation part (100), determine whether the preliminary pixel data satisfies the longitudinal data reuse rule of the Z-shaped data stream, and if so, perform longitudinal data movement processing on the preliminary pixel data.

7. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 1, characterized in that, The calculation section (300) includes: The calculation module (310) has a built-in multiply-accumulate array, the size of which is determined by the loop unrolling variable; the loop unrolling variable expresses the parallelism of the calculation; the calculation module (310) is used to perform convolution calculation on the target pixel data and the target weight data in the multiply-accumulate array according to the convolution layer parameters to obtain the output pixel data.

8. An architecture supporting convolution kernel computation of arbitrary size and shape according to claim 1, characterized in that, The output data rearrangement and storage section (400) includes: The output data arrangement module (410) is used to rearrange the output pixel data according to the data reading requirements of the next layer calculation; An output buffer (420) is used to store the output pixel data rearranged by the output data arrangement module (410).

9. The architecture supporting convolution kernel computation of arbitrary size and shape according to claim 1, characterized in that, The control unit (500) includes: Layer-by-layer control module (510) is used to control the input data preparation part (100), the zigzag data stream preparation part (200), the calculation part (300) and the output data rearrangement and storage part (400) to realize layer-by-layer convolution calculation; The layer-by-layer control module (510) includes a sub-core control module (511). The sub-core control module (511) controls the reading order of input pixels and weight data to ensure that the data flow of each sub-core still satisfies the Z-shaped data flow pattern under the condition of a large convolution kernel size and stride.

10. An architecture supporting convolution kernel computation of arbitrary size and shape according to claim 5, characterized in that, The specific implementation of the Z-shaped data stream preparation part (200) includes: Send the prepared pixel data to the corresponding register; Based on the input pixel reuse pattern of the horizontal movement of the convolution window during convolution calculation, the designed Z-shaped data flow provides opportunities for horizontal data reuse. The prepared pixel data parallel to the row direction of the input feature map is assigned to the adjacent register array on the left or right in the next cycle to achieve data reuse. The direction of assignment is determined by the parity of the number of rows of the corresponding weight data in the convolution kernel. Odd-numbered rows go to the left and even-numbered rows go to the right. By introducing computation at the kernel mapping inflection point, the opportunity for vertical data reuse is increased. The prepared pixel data, which is parallel along the column direction of the input feature map, is assigned to the register array on the left in the next cycle to achieve reuse. The decoupling of horizontal and vertical data reuse enables the architecture to support the computation of convolution kernels of arbitrary size and shape. When supporting larger convolution kernel sizes or strides, the convolution kernel is divided or split into several sub-kernels and controlled so that the computational data flow corresponding to each sub-kernel still satisfies the Z-shaped data flow pattern. The final convolution computation result is obtained by accumulating the computation results of different sub-kernels.

Images