Computing devices, methods for performing convolution operations using computing devices, and related products

Abstract

This disclosure discloses a computing device, a method for performing convolution operations using the computing device, and related products. The computing device may be included in a combined processing unit, which may further include an interface device and other processing units. The computing device interacts with the other processing units to jointly complete user-specified computational operations. The combined processing unit may also include a storage device connected to both the computing device and the other processing units to store data from them. The solution disclosed here optimizes convolution operations, improving computational processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a computing device configured to perform convolution operations, a method for performing convolution operations using the computing device, a chip, and a board. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Neural networks are among the most critical technologies in artificial intelligence and deep learning, with Convolutional Neural Networks (CNNs) being the most important type. The most crucial computation in CNNs is the convolution operation within the convolutional layers (Conv layers). The function of convolutional layers is to extract features from input data. Through multiple convolutions, complex features can be extracted to ensure the network has sufficient expressive and generalization capabilities. Neural network models contain numerous convolution operations of various types, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. When neural network models are applied to different fields, such as speech recognition, machine translation, and image processing, the corresponding input feature maps and the dimensions of the weights may vary. To fully utilize the hardware advantages of deep learning processors, optimization is needed for convolution operations of different scales and / or types to improve the computational performance of the neural network model. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes a computing device in several aspects that, by folding the width dimension of the input feature map, enables data of various dimensional sizes to be adapted to the hardware of convolution operations, thereby improving the computational efficiency of convolution operations. The convolution operations in embodiments of this disclosure can be operations in various neural network models applicable to various fields, such as image processing, speech processing, text processing, etc., and these processes may include, but are not limited to, recognition and classification.

[0005] In a first aspect, embodiments of this disclosure provide a computing device configured to perform convolution operations, the computing device comprising: a storage circuit storing an input feature map and convolution kernels, wherein the convolution kernels are stored in blocks according to the output channel Co dimension for allocation to different slave processing circuits; a main processing circuit configured to align the input channel Ci dimension of the input feature map to an input channel splitting granularity Pci, and to combine the data of each splitting granularity with data of a width dimension Ws times to form an input feature data row and broadcast it to a plurality of the slave processing circuits to perform the convolution operation, wherein Pci*Ws = M, M is the amount of data processed by the hardware in a single operation; and a plurality of the slave processing circuits configured to perform convolution operations on the broadcast input feature map and the corresponding convolution kernels allocated to the slave processing circuits, respectively.

[0006] In a second aspect, embodiments of this disclosure provide a chip that includes a computing device according to any of the embodiments of the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip of any of the embodiments of the second aspect above.

[0008] In a fourth aspect, embodiments of this disclosure provide a method for performing convolution operations using a computing device according to any of the embodiments of the first aspect.

[0009] By utilizing the computing device, chip, board, and method for performing convolution operations as described above, the scheme of this disclosure applies different width dimension folding schemes to input feature maps of different dimensions to adapt to the processing capabilities of the hardware computing device, thereby fully utilizing the parallel processing capabilities of multiple processing circuits and effectively improving the computational efficiency of convolution operations. Furthermore, since the width dimension is adjacent to the input channel dimension at the lowest storage dimension, width dimension folding does not require additional software slewing, simplifying the processing procedure and instruction overhead. In some embodiments, since the input channel dimension is aligned by hardware during data path transmission, no software alignment is required, reducing the amount of memory access I / O to the storage circuits. In addition, the convolution operation scheme of this disclosure can support convolution strides of arbitrary size in the height direction. The convolution operation scheme of this disclosure is applicable to convolution operations of any input channel dimension Ci. Other advantages and effects will become readily apparent from the following detailed description in conjunction with the accompanying drawings. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0012] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0013] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0014] Figure 4 This illustrates an exemplary example of the convolution operation principle that can be applied to the embodiments disclosed herein;

[0015] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0016] Figures 6a-6c Several examples of data width dimension folding according to embodiments of this disclosure are shown;

[0017] Figures 7a-7c Several exemplary zero-plugging alignment processes according to embodiments of this disclosure are illustrated;

[0018] Figure 8 A schematic diagram of the convolution kernel storage method according to an embodiment of this disclosure is shown;

[0019] Figure 9 A schematic splitting method of the output feature map according to an embodiment of this disclosure is shown;

[0020] Figures 10a-10d A schematic diagram illustrating the operation process of a convolution operation scheme according to an embodiment of this disclosure is shown.

[0021] Figure 11 A schematic diagram illustrating the logic for writing and outputting computation results according to an embodiment of this disclosure is shown;

[0022] Figure 12 A schematic diagram illustrating the operation process of a convolution operation scheme according to another embodiment of this disclosure is shown;

[0023] Figure 13 A schematic diagram illustrating the operation process of a convolution operation scheme according to yet another embodiment of this disclosure is shown;

[0024] Figure 14 A schematic diagram illustrating the operation process of a convolution operation scheme according to another embodiment of the present disclosure is shown.

[0025] Figure 15 This diagram illustrates the computational process of a convolution operation scheme according to yet another embodiment of this disclosure; and

[0026] Figure 16 A schematic diagram illustrating the writing and output logic of computation results according to an embodiment of this disclosure is shown. Detailed Implementation

[0027] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0029] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0030] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0031] Exemplary hardware environment

[0032] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0033] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0034] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0035] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0036] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0037] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0038] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0039] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0040] Figure 3 The diagram shows the internal structure of the processing core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0041] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0042] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0043] The storage module 33 is used to store or move relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. The NRAM 331 stores the input neurons, output neurons, and intermediate results after computation; the WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; the DMA 333 is connected to the DRAM 204 via bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204.

[0044] Exemplary convolution operation types

[0045] Based on the aforementioned hardware environment, in one aspect, this disclosure provides a computing device configured to perform convolution operations, thereby optimizing convolution operations, for example, in neural network models. Convolutional layers in a neural network model can perform convolution operations by applying convolution kernels (also called filters, weights, etc.) to an input feature map (also called input data, neurons, or input neurons) to extract features. A convolutional layer may contain multiple convolution kernels, each element of which corresponds to a weight coefficient and a bias.

[0046] Neural network models may contain various convolutional layers, such as convolutional layers that perform forward, regular 3D convolutional operations, and deconvolutional layers that perform depthwise convolutional operations. In reverse training, it may be necessary to perform inverse depthwise convolutional operations or cross-product convolutional operations. The embodiments disclosed herein are primarily optimized for regular 3D convolutional operations, but can also be applied to other types of convolutional operations without conflict.

[0047] In conventional 3D convolution operations, assuming the input feature map tensor shape in the convolutional layer is represented by X[N Hi Wi Ci], the tensor shape of the convolutional kernel is represented by K[Co Kh Kw Ci], and the output is Y[N HoWo Co], then the simplified mathematical formula for convolution operations can be expressed as follows:

[0048] Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0049] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, and sh and sw are the strides in the length and width directions. The formula ignores the bias, padding, and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N and C dimensions. The forward computation of the neural network model is independent in the N dimension, but fully connected in the C dimension. When the convolution kernel works, it scans the input features with a certain stride, performs matrix element-wise multiplication and sums the bias within the convolution window. In conventional 3D convolution operations, the results of the alignment multiplications in the H, W, and Ci directions are accumulated, hence the name 3D convolution. However, this 3D convolution has a constraint: the Ci dimension of the convolution kernel is equal to the Ci dimension of the input feature map, so the convolution kernel does not slide in the Ci direction, making it a pseudo-3D convolution. For simplicity, the above convolution operation is called 3D convolution operation.

[0050] Figure 4 An example of a conventional 3D convolution operation principle that can be applied to embodiments of this disclosure is shown.

[0051] The figure exemplifies a four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N Hi×Wi×Ci 3D rectangles 410. The figure also exemplifies a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co Kh×Kw×Ci 3D convolution kernels 420. The convolution result of the input data X and the convolution kernel K yields the output data Y, which is a four-dimensional data of size [N Ho Wo Co], which can be represented as N Ho×Wo×Co 3D rectangles 430.

[0052] The figure also shows a specific example of a convolution operation, where the input data is a 6×6×3 input feature map 440, omitting the N dimension; the convolution kernel is a 3×3×3 stereo convolution kernel 450, targeting a single Co; and the output data is a 4×4 output feature map 460. The specific operation process is as follows:

[0053] The convolution kernel 450 scans across the input feature map 440a with a certain stride, performs element-wise matrix multiplication on the input features within the convolution window 470, and then sums the bias values. That is, the value at each position in the output feature map 460 is obtained by performing a two-dimensional convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) in the output feature map 460 (i.e., the convolution output point) is obtained by performing a two-dimensional convolution operation between the convolution window 470 (framed by the black cube in the input feature map) and the 3D convolution kernel 450, resulting in three values, which are then summed to obtain the final value.

[0054] To obtain outputs at other locations, the position of the convolution kernel 450 can be moved on the input feature map 440, which means moving the convolution window of the output point. In the example in the figure, the convolution stride (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0,1) or (1,0) on the output feature map 460a can be obtained respectively.

[0055] As described above, a convolutional layer in a neural network contains N sets of input feature maps, each containing Hi × Wi × Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output feature map contains Ho × Wo × Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation also involves the convolution stride (Sx, Sy), the size of which affects the size of the output feature map.

[0056] In this paper, the input feature map, input data, neuron, or input neuron can be used interchangeably; the convolution kernel, filter, or weights can be used interchangeably; and the output feature map, output data, or output neuron can be used interchangeably. Furthermore, the H (height) and Y dimensions can be used interchangeably, as can the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In the embodiments disclosed herein, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window.

[0057] Exemplary computing device

[0058] In this disclosed embodiment, a master-slave computing device can be used to perform the above-described convolution operation. Furthermore, different data paths can be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0059] Figure 5 A schematic structural block diagram of a computing device 500 according to an embodiment of this disclosure is shown. It can be understood that this structure can be considered as... Figure 3 The detailed internal structure of the processing module of a single processing core can also be viewed as a refinement of multiple processing cores. Figure 3 The diagram shows a functional partitioning based on the processing core's computational modules. Figure 5 As shown, the computing device 500 of this disclosure embodiment can be configured to perform various types of convolution operations, and may include a main processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520. Sixteen slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will understand that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and this disclosure embodiment is not limited in this respect.

[0060] The master processing circuit and slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections. In different application scenarios, the connection between multiple slave processing circuits can be either a hard connection arranged by hardwired lines or a logical connection configured according to, for example, microinstructions, to form a topology of various slave processing circuit arrays. The embodiments disclosed herein are not limited in this respect. The master processing circuit and slave processing circuits can cooperate with each other to achieve parallel processing.

[0061] To support computational functions, the main processing circuit and the slave processing circuit can include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit is responsible for the core computations of deep learning algorithms, such as matrix multiplication and convolution.

[0062] The processing circuit can, for example, perform intermediate operations on the corresponding data in parallel according to the operation instructions to obtain multiple intermediate results, and then transmit the multiple intermediate results back to the main processing circuit.

[0063] By configuring the computing device 500 into a master-slave structure (e.g., a master-multiple-slave structure, or a multi-master-multiple-slave structure, which is not limited in this disclosure), for forward computation instructions, the data can be split according to the computation instructions, thereby enabling multiple slave processing circuits to perform parallel computation on the computationally intensive parts to improve computation speed, save computation time, and thus reduce power consumption.

[0064] In some embodiments disclosed herein, by utilizing different data paths to transmit the input feature map and weights, multiple reuse methods of the input feature map and weights can be supported, thereby reducing the data throughput during computation and improving processing efficiency.

[0065] Specifically, the computing device 500 may further include a first storage circuit 530 and a second storage circuit 540 for storing data transmitted via different data channels, respectively. Optionally, the first storage circuit 530 and the second storage circuit 540 may be two storage blocks formed by partitioning the same memory, or they may be two independent memories; no specific limitation is made here.

[0066] The first storage circuit 530 can be used to store multicast data, meaning that the data in the first storage circuit will be transmitted to multiple slave processing circuits via a broadcast bus, and these slave processing circuits will receive the same data. It can be understood that broadcasting and multicasting can be implemented via a broadcast bus. Multicast refers to a communication method that transmits a single data set to multiple slave processing circuits; while broadcasting is a communication method that transmits a single data set to all slave processing circuits, and is a special case of multicast. Since both multicast and broadcasting correspond to one-to-many transmission methods, this document does not specifically distinguish between the two; broadcasting and multicast can be collectively referred to as multicast, and those skilled in the art can understand their meaning from the context.

[0067] The second storage circuit 540 can be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0068] By providing a first storage circuit and a second storage circuit respectively, it is possible to support the transmission of data to be processed in different transmission modes, thereby reducing data throughput by multiplexing multicast data among multiple slave processing circuits.

[0069] In some embodiments, the input feature map can be determined as multicast data and stored in a first storage circuit to transmit the data via broadcast to multiple scheduled slave processing circuits during computation. Correspondingly, the convolution kernel can be determined as distribution data and stored in a second storage circuit. This distribution data can be distributed to the corresponding slave processing circuits before computation.

[0070] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of this disclosure is also shown. As shown, each slave processing circuit 520 may include multiple arithmetic circuits CU 521, a first buffer circuit 522, and a second buffer circuit 523. Four arithmetic circuits CU0 to CU3 are shown in the figure. Those skilled in the art will understand that the number of arithmetic circuits may be more or less, depending on the specific hardware configuration, and the embodiments of this disclosure are not limited in this respect.

[0071] In some embodiments, the first buffer circuit 522 can be used to buffer the weights or input feature maps allocated to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to buffer the input feature maps or weights allocated to the slave processing circuit. Both buffer circuits are used to select data for computation. The data in the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540; correspondingly, the data in the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computation circuit CU 521 or broadcast to all CUs 521 within the slave processing circuit 520 during computation.

[0072] Each arithmetic circuit CU 521 is used to perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit and data rows selected from the second buffer circuit, respectively, in each arithmetic cycle.

[0073] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be processed in different transmission modes, thereby reducing data throughput by reusing data as much as possible among multiple processing circuits within a single slave processing circuit.

[0074] The processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each arithmetic circuit CU 521.

[0075] Understandable, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but depending on the configuration, the storage circuits and processing circuits can also be combined into a single module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, while the second storage circuit 540 can be shared by multiple slave processing circuits 520, with each slave processing circuit allocated an independent storage area to accelerate access. This disclosure does not limit the embodiments in this respect. Furthermore, in this computing device, the main processing circuit and slave processing circuits can belong to different modules of the same processor or chip, or they can belong to different processors; this disclosure also does not limit this in this respect.

[0076] Exemplary Input Feature Map W Folding Scheme

[0077] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is typically allocated in contiguous storage space, meaning that multidimensional data can be unfolded one-dimensionally and stored sequentially in memory. For example, in this disclosed embodiment, the initial input feature map can be stored sequentially in a low-dimensional priority manner (here, C / Ci is the lowest dimension). Adjacent dimensions refer to dimensions that are immediately adjacent in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. When the storage order is consistent with the dimension order, adjacent dimensions are located consecutively in memory. Here, W and Ci are adjacent, and their data is also consecutively stored in memory.

[0078] In intelligent processors, due to computational power requirements and considerations of area and power consumption, the main hardware operation unit is the vector multiply-accumulate unit. Implementing support for various convolution algorithms in hardware design essentially involves maximizing the extraction of multiply-accumulate operations from the algorithms and implementing them in on-chip RAM (such as...) via data paths. Figure 3 It efficiently exchanges input and output data for multiplication and addition operations between NRAM, WRAM, etc. and the arithmetic unit.

[0079] Hardware storage operates in rows (cache lines). Read, write, and computation operations are most efficient when aligned to entire rows. Therefore, to fully utilize bandwidth and meet the throughput requirements of the arithmetic unit array, data usually needs to be vectorized and aligned. Artificial intelligence chips are typically designed with the Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above, where data along the Ci dimension is continuous. Therefore, vectorization alignment requires the Ci dimension to be aligned to a specified value, such as alignment value M, so that data can be accessed in units of this alignment value M. M can also be called the maximum number of operations per hardware operation. Depending on the hardware design, M can have different values, such as 64 bits, 128 bits, 256 bits, 512 bits, etc. Typically, the input port size of the arithmetic unit array is also related to M. For example, in the case of symmetrical input data bit width, the input port size of the arithmetic unit array is usually twice M, meaning it processes input feature map data and weight data of alignment value M in one operation. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0080] When the Ci dimension of the input feature map is small, or when the remainder obtained by dividing Ci by M is small (e.g., less than the size of a cache line), the Ci dimension needs to be padded to a single line of data (e.g., 512 bits), i.e., filled with invalid data 0. This padding causes a large amount of redundant computation, resulting in wasted resources and reduced computational efficiency.

[0081] A small convolution scheme suitable for cases with a small channel C has been proposed, in which the computational data is split into split units and stored in a transformed dimensional order. The amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, thereby performing computational processing on a unit-by-unit basis, which can fully utilize the hardware's computing power and avoid or reduce invalid computations.

[0082] However, in this small convolution scheme, both the input feature map and the convolution kernel need to be pre-processed by software through block division and dimensionality transformation, and the output feature map also needs to be correspondingly processed by software through block division and dimensionality transformation, which undoubtedly increases the complexity of the software. Furthermore, alignment processing is also required during these block division and dimensionality transformation processes. Moreover, these small convolution schemes only support convolution operations with a stride of 1 in both the width and height directions.

[0083] In view of this, in order to further optimize convolution operations and reduce software complexity, this disclosure provides a width-dimensional folded convolution scheme, abbreviated as Wfold, which compensates the data of the width W dimension, which is continuous with the input channel Ci dimension of the input feature map, to the Ci dimension only when needed, thus eliminating the need for software to perform data block and dimension transformation processing.

[0084] Specifically, in some embodiments, a computing device is provided, configured to perform convolution operations. The computing device includes: a storage circuit storing an input feature map and convolution kernels, wherein the convolution kernels are stored in blocks according to the output channel Co dimension for allocation to different slave processing circuits; a main processing circuit for aligning the input channel Ci dimension of the input feature map to an input channel splitting granularity Pci, and broadcasting the data of each splitting granularity, combined with data of a width dimension Ws times the width dimension, to multiple slave processing circuits to perform convolution operations, where Pci*Ws = M, and M is the amount of data processed in a single hardware operation; and multiple slave processing circuits for performing convolution operations on the broadcast input feature map and the corresponding convolution kernels allocated to the slave processing circuits, respectively.

[0085] In some embodiments, the input channel splitting granularity Pci can be determined based on the size of the input channel dimension Ci of the input feature map and the amount of data M processed by the hardware in a single operation. Then, the folding factor Ws of the width dimension W of the input feature map can be determined based on the splitting granularity Pci. In some embodiments, Ws = M / Pci. It is understood that the convolution scheme of this disclosure embodiment, by splitting the Ci dimension according to the splitting granularity, can be suitable for any Ci size. Furthermore, it is understood that the maximum splitting granularity Pci does not exceed the hardware's single-operation alignment value M (or baseline alignment value, the amount of data processed by the hardware in a single operation). Therefore, under different value ranges of Ci, a suitable Pci can be selected, and by filling the adjacent W dimensions with data in the Ci dimension, the alignment requirements for the Ci dimension can be reduced.

[0086] In some embodiments, the input channel splitting granularity Pci can be selected as M / 2. n n = 0, 1, 2, ..., thus facilitating storage from the second lowest dimension W in 2... n Fold the data to the lowest storage dimension Ci. Table 1 shows several exemplary folding schemes corresponding to the input channel splitting granularity Pci, assuming M = 64B.

[0087] Split granularity (Pci) 4B 8B 16B 32B 64B Ws (W-fold) 16 8 4 2 1

[0088] Table 1

[0089] As can be seen from Table 1, the smaller the input channel splitting granularity, the more parts Wi supplies in the Ci direction, and the larger the alignment constraint on Wi, which needs to satisfy Wi / Ws≥1.

[0090] This is understandable, although theoretically the granularity of the split can be M / 2. n However, considering factors such as the requirements for the W dimension, instruction overhead, and the actual range of Ci values ​​when the splitting granularity is too small, it is advisable to choose only M / 2. nA subset of the values ​​in the table are used as alternative splitting granularities. In the example where M = 64B, alternative splitting granularities could include, for example, 64B, 32B, 16B, and 8B.

[0091] Different splitting granularities can be applied to different computational scenarios, thereby achieving varying degrees of performance optimization. Specifically, in some embodiments, the input channel splitting granularity Pci can be selected as follows:

[0092] Align the lowest storage dimension Ci of the input feature map to each of the candidate splitting granularities; and

[0093] Taking into account the alignment padding amount and the size of the corresponding splitting granularity for each candidate splitting granularity, a suitable splitting granularity is selected, such as a candidate splitting granularity with an alignment padding amount within a predetermined range and as large as possible, as the Pci.

[0094] For example, when the alignment fill amount is the same, a larger split granularity is preferred; or when the alignment fill amount is different, a split granularity with the smallest alignment fill amount is selected; or when the alignment fill amounts are not significantly different (e.g., within a predetermined range, such as no more than 16B), a larger split granularity is preferred.

[0095] Although the rules for selecting the input channel splitting granularity Pci are listed above, these rules are merely preferred embodiments for selecting the optimal input channel splitting granularity best suited to the current Ci value. The application of these rules is described below with several examples. All examples assume M = 64B, and alternative splitting granularities include 64B, 32B, 16B, and 8B.

[0096] In one example, assuming Ci = 48B, alignment to 8B and 16B requires no zero padding, while alignment to 32B and 64B requires 16B padding. In this case, the larger of the splitting granularities that does not require zero padding can be preferred as Pci, i.e., 16B.

[0097] In another example, assuming Ci = 28B, alignment to 8B, 16B, and 32B all require padding with 4B zeros, while alignment to 64B requires padding with 36B zeros. In this case, a smaller alignment padding amount and a larger splitting granularity can be preferred as Pci, i.e., 32B.

[0098] In another example, assuming Ci = 49B, aligning to 8B requires padding with 7B zeros, while aligning to 16B, 32B, and 64B all require padding with 15B zeros. In this case, the alignment padding amount differs by only 8B, which is within an acceptable range. Therefore, the larger splitting granularity of 64B can be preferred.

[0099] Figures 6a-6cSeveral examples of data width dimension folding according to embodiments of this disclosure are shown. In these examples, it is also assumed that M = 64B.

[0100] like Figure 6a As shown, when the input channel splitting granularity Pci = 16B, the W dimension needs to be folded by a factor of 4. That is, the shape of a data row is Wi*Ci = 4 × 16B. When the size of the Ci dimension exceeds 16B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 48B, its data will be split into 3 data rows. The figure uses rounded rectangles to show the data included in each of the 3 data rows; here, 3 can also be referred to as the number of split blocks in the Ci dimension.

[0101] like Figure 6b As shown, when the input channel splitting granularity Pci = 32B, the W dimension needs to be folded by a factor of 2. That is, the shape of a data row is Wi*Ci = 2 × 32B. Similarly, when the size of the Ci dimension exceeds 32B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 96B, the data thereon will be split into 3 data rows. Only a single data row is shown in the figure.

[0102] like Figure 6c As shown, when the input channel splitting granularity Pci = 64B, the W dimension needs to be folded by 1, meaning it doesn't need to be folded. In this case, the shape of a data row is Wi*Ci = 1 × 64B. Similarly, when the size of the Ci dimension exceeds 64B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 128B, the data therein will be split into two data rows. The figure only shows a single data row.

[0103] As mentioned above, in some embodiments, Figure 5 The main processing circuit 510 can determine the input feature map as multicast data and store it in the first storage circuit 530, so that the data can be transmitted to multiple scheduled slave processing circuits via broadcast during the operation. As can be seen from the width folding scheme described above, since WC is a continuous dimension, the format of the input data does not need to be divided and dimensionality transformed, and the original input data format HWC can be received directly. Therefore, the input feature map can be stored in the first storage circuit 530 in its original format (e.g., HWC).

[0104] When the input feature map is read from the first storage circuit 530 and broadcast to multiple slave processing circuits, the aforementioned alignment process can be performed. That is, during the transmission from the first storage circuit to the buffer circuit (e.g., the first buffer circuit) within the slave processing circuit, the main processing circuit 510 can control the alignment process of the Ci dimension to align to the determined input channel splitting granularity Pci, and then fold the corresponding number of Wi dimension data to form a data row, and broadcast it to the slave processing circuit with a data row as the smallest granularity.

[0105] Different data read / write schemes can be used to achieve the data transfer process from the first storage circuit to the buffer circuit within the processing circuit.

[0106] In some embodiments, input feature map data can be read directly from the first storage circuit and then continuously written to the buffer circuit of the processing circuit. During the read-to-write process, padding to the input channel split granularity Pci is performed as needed. In these embodiments, depending on the number of split blocks in the Ci dimension, either continuous or intermittent (non-continuous) reads can be adopted. Note that "continuous" or "intermittent" in this document refers to data being continuous or intermittent in storage address.

[0107] In one example, when the number of partitions along the Ci dimension is 1, meaning the Ci dimension is not partitioned, Ci ≤ Pci, and a W contains only 1 Pci. When Ws Ws need to be read, these Ws are contiguous. Therefore, in this case, Ws Ws can be read at once with a single read request, simplifying the command. Of course, multiple read requests can also be used to read contiguous data from Ws Ws.

[0108] Figure 7a An exemplary data read / write scheme and possible zero-plugging alignment are shown as an example according to this disclosure.

[0109] like Figure 7a As shown, assuming the input data has a shape of 4×9×14B (HWC) and the determined input channel splitting granularity Pci is 16B, then the number of split blocks in the Ci dimension is 1. Each time, Ws = 4 Ws need to be merged into a single data line for output, where each W needs to be padded with zeros in the Ci direction to reach 16B. That is, 4 Ws are read from the first storage circuit, each W is padded with 2B of zeros, thus merging into a 4×16B = 64B data line broadcast to the slave processing circuit.

[0110] In this example, since the number of split blocks in the Ci dimension is 1, the 4Ws read are continuous. They can be read out all at once with a single read request, or they can be read out as continuous data in the 4Ws with multiple read requests.

[0111] In another example, when the number of partition blocks on the Ci dimension is not 1, that is, when the Ci dimension is partitioned into multiple partition granularities Pci, Ci > Pci. Since a data row consists of Ws * Pci, it is necessary to read the Pci on each W at intervals. The reading interval is 1 * Ci(WCi), which is 1 W. Each time the data volume of Pci is read, Ws reads are required. In other words, Ws reads are performed, each time reading the data volume of Pci on 1 W at equal intervals. In this case, data reading can be achieved through Ws read requests, with each read request reading the data volume of Pci on the corresponding W.

[0112] Figure 7b An exemplary data read / write scheme and possible zero-plugging alignment are shown in another example according to this disclosure.

[0113] like Figure 7b As shown, assuming the input data has a shape of 4×9×47B (HWC) and the determined input channel splitting granularity Pci is 16B, then each time, a portion of the data from Ws = 4 Ws needs to be merged into a single data line for output. Each W needs to be padded with zeros in the Ci direction to reach 16B*3 = 48B. This requires interval reading, which can be achieved through four read requests. Specifically, four read requests can be sent, reading the first Pci = 16B data segment from each W at equal intervals. That is, the first request reads the data B0 of the first Pci from W0 in the first storage circuit, the second request reads the data B0 of the first Pci from W1, the third request reads the data B0 of the first Pci from W2, and the fourth request reads the data B0 of the first Pci from W3. The data from each Pci of these four Ws is merged into a data line of 4*Pci = 64B and stored in the buffer circuit. Next, four more read requests are sent, reading the second Pci=16B data block from each W at equal intervals. Finally, four more read requests are sent, reading the last data block from each W at equal intervals. At this point, alignment processing can be performed, inserting 1B of zeros into each data block, thus merging them into a 4×16B=64B data line which is then broadcast to the slave processing circuit.

[0114] In other embodiments, when data needs to be read from the first storage circuit using an interval reading method, considering that the first storage circuit is typically RAM, implementing interval reading is difficult and costly. Therefore, an intermediate storage circuit can be provided to simplify the implementation. The intermediate storage circuit can be implemented using registers, thereby easily enabling interval or discontinuous reading.

[0115] Specifically, in these embodiments, a certain amount of continuous input feature map data can first be read from the first storage circuit, and the read data can be written to a transfer storage circuit. Then, data can be read intermittently from the transfer storage circuit and continuously written to the buffer circuit of the slave processing circuit. During the reading from the transfer storage circuit to the write buffer circuit, padding processing to align to the input channel splitting granularity Pci can be performed as needed. The continuous data volume read from the first storage circuit can be Ws Ci, that is, Ws*Ci of continuous data is read and stored on the transfer storage circuit. Then, data can be read intermittently from the transfer storage circuit and stored on the buffer circuit of the slave processing circuit in the same manner as in the previous example.

[0116] Figure 7c An exemplary data read / write scheme and possible zero-padding alignment are shown as yet another example according to this disclosure.

[0117] like Figure 7c As shown, still assuming the shape of the input data is 4×9×47B (HWC), and the determined input channel splitting granularity Pci is 16B, then each time Ws = 4 Ws need to be merged into a data line for output, where each W needs to be padded with zeros in the Ci direction to 16B*3 = 48B.

[0118] In this example, a certain number of consecutive input feature map data can be read from the first storage circuit, and the read data can be written to a transfer storage circuit. Specifically, four 1*47B data points can be read sequentially, for example, through four read requests. First, read continuously in the Ci direction until one Ci is read, then move 1 in the W direction and read the next Ci, until four consecutive W data points are read. In one implementation, four read requests can be sent separately: the first request reads 47B (Ci) data on W0 from the first storage circuit; the second request reads 47B data on W1; the third request reads 47B data on W2; and the fourth request reads 47B data on W3. The 47B data points read from each of the four Ws are then written to the transfer storage circuit.

[0119] Next, 3*(4*16B) of data are retrieved from the intermediate storage circuit and fed into the buffer circuit of the processing circuit for convolution operation. The process from 4*(1*47B) to 3*(4*16B) is similar to... Figure 7b An example of this is intermittent reading on the intermediate storage circuit. This will not be repeated here.

[0120] Thus, the storage format of the input feature map and the folding process via the data channel in this disclosure embodiment have been described above.

[0121] Exemplary convolution kernel storage

[0122] The calculation of convolution requires each input feature map to perform multiplication and addition operations with each convolution kernel of Co, thereby outputting Co output feature maps. However, it is not certain that all scales of convolution kernels and input feature maps can be stored simultaneously in the on-chip space. Therefore, for hardware, there are a series of operations to repeatedly load input feature data or weight data. How to balance the repeated loading of input feature data or weight data will have a certain impact on the computational efficiency. In actual operations, in order to reduce frequent off-chip memory accesses, different reuse methods can be adopted according to the scale characteristics of the data participating in the operations.

[0123] According to the convolution operation principle described above, the operation results in the Co dimension do not need to be accumulated. Therefore, the operations on different Cos can be carried out relatively independently on different operation circuits. That is, different convolution kernels of Co can be allocated on different operation circuits, and the same input feature map is used for operations. At this time, the input feature map is reused among these operation circuits, and the reuse times Rn = Ns, where Ns is the number of operation circuits.

[0124] In some embodiments of the present disclosure, the Co value assigned to each slave processing circuit for processing can be determined based on the output channel Co dimension size of the convolution kernel and the number Ns of schedulable slave processing circuits.

[0125] To simplify the scheduling of slave processing circuits, in some embodiments, the output channel alignment value Aco can be determined according to the size of the output channel dimension Co of the convolution kernel; then, according to the alignment value Aco, the convolution kernel can be block-stored according to the Co dimension for the scheduled slave processing circuits to load the corresponding weight blocks. In some embodiments, the output channel alignment value Pco can be selected as a multiple of the number Ns of schedulable slave processing circuits, so as to facilitate the average distribution of Co to Ns slave processing circuits. For example, each slave processing circuit processes convolution kernels with different Aco / Ns Co values at an interval of Ns.

[0126] Specifically, the preferred output channel alignment value Aco can be determined according to the value range of Co. For example, the following rules can be followed:

[0127] When Ns*(m - 1) < Co ≤ Ns*m, where m = 1, 2, …, Aco = Ns*m.

[0128] As mentioned earlier, in some embodiments, the convolution kernel can be defined as distribution data and stored in the second storage circuit 540 for distribution to the corresponding slave processing circuit or for reading by the slave processing circuit before computation. The second storage circuit 540 can be shared by multiple (e.g., Ns) slave processing circuits 520, with each slave processing circuit allocated an independent storage area. This ensures that the data required for computation by each slave processing circuit only needs to be read from its corresponding storage area, accelerating memory access. When the convolution kernel is partitioned and stored according to the Co dimension, the convolution kernel corresponding to the Co value allocated to a slave processing circuit can be stored in the corresponding storage area of ​​the second storage circuit. Since the Co dimension is the highest storage dimension of the convolution kernel, partitioning and storing along the Co dimension does not require dimension transformation or other processing; the convolution kernel data corresponding to the Co value can be directly stored in the second storage circuit in its original format (e.g., KhKwCi).

[0129] Figure 8 A schematic diagram of a convolution kernel storage method according to an embodiment of this disclosure is shown. The diagram exemplarily illustrates 16 storage regions 800-815 allocated to, for example, 16 slave processing circuits SL0-SL15 (Ns = 16). Each storage region stores the convolution kernel corresponding to the Co value to be processed by that slave processing circuit.

[0130] As mentioned earlier, the Co dimension of the convolution kernel is aligned to Aco. When Ns = 16, Aco can be 16, 32, 48, 64, etc. Therefore, the convolution kernels are divided according to Co and stored in various storage regions of the second storage circuit to be allocated to the corresponding SL. The storage region corresponding to each SL stores the convolution kernels assigned to it with different Co values. In one example, consecutive Co values ​​are sequentially allocated to 16 SLs one by one (i.e., in units of 1), and the allocation starts again after one round. For example, the figure shows that the convolution kernels with Co = 0 to 15 are stored sequentially in 16 storage regions 800 to 815; the next convolution kernels with Co = 16 to 31 are stored sequentially in 16 storage regions 800 to 815; and so on. Thus, after each round of operation, the Co dimension of the operation results output from the processing circuit is continuous. In another example, the convolution kernels can also be divided into blocks according to Co values, with each SL processing one block, and each block including convolution kernels with consecutive Aco / Ns Co values. For example, assuming Aco = 64, convolutional kernels with Co = 0–3 are stored in storage area 800 allocated to SL0, convolutional kernels with Co = 4–7 are stored in storage area 801 allocated to SL1, convolutional kernels with Co = 8–11 are stored in storage area 802 allocated to SL2, and so on, until convolutional kernels with Co = 60–63 are stored in storage area 815 allocated to SL15. Thus, the Co dimension processed by each SL is continuous.

[0131] As can be seen from the Co partitioning above, each slave processing circuit may need to process one or more Co-value convolutional kernels. When processing multiple Co-values, the input feature map processed by each slave processing circuit can be further reused for these multiple Co-value convolutional kernels, with a maximum reuse count rn = Aco / Ns, where rn represents the number of times the input feature map is reused within a single slave processing circuit. Factors such as hardware buffer space limitations (e.g.) can be considered. Figure 5 The size of the first and second buffer circuits in the circuit determines the number of times the input feature map can be multiplexed within a single processing circuit.

[0132] The above describes how convolutional kernels are aligned along the Co dimension and stored in blocks in the second storage circuit. Similar to the input feature map, each Co value of the convolutional kernel also undergoes similar alignment along the Ci dimension.

[0133] In some embodiments, the alignment process described above can be performed when the convolution kernel is read from the second storage circuit and distributed to the corresponding slave processing circuit. That is, during the transfer from the second storage circuit to the buffer circuit (e.g., the second buffer circuit) within the slave processing circuit, alignment processing of the Ci dimension of the convolution kernel can be performed to align it to the previously determined input channel splitting granularity Pci. Unlike the input feature map, the convolution kernel does not need to be folded in the W dimension, but is instead expanded by copying according to the folding factor Ws, as can be seen in the subsequent description of the convolution operation process.

[0134] A single exemplary convolution operation process within the processing circuit

[0135] When the input feature map is broadcast to the scheduled slave processing circuits, and the convolution kernels are distributed to the corresponding slave processing circuits, each slave processing circuit can simultaneously perform convolution operations on the corresponding data of the input feature map and the convolution kernels. Then, the main processing circuit can concatenate the operation results returned by multiple slave processing circuits according to the convolution width folding scheme to obtain the output feature map of the convolution operation between the input feature map and the convolution kernels. Specifically, multiple computational circuits (CUs) and various buffer circuits in the slave processing circuits can be utilized (see...). Figure 5 The specific convolution operation is performed by a buffer circuit. Depending on the size of the buffer circuit inside the processing circuit and the computing power limitations of the computing circuit, multiple operation cycles are usually required to complete the required operation in each round of operation.

[0136] In some embodiments, the first buffer circuit can be used to cache the input feature map from the first storage circuit; correspondingly, the second buffer circuit can be used to cache the convolution kernel from the second storage circuit. Each arithmetic circuit CU can perform bitwise multiplication-accumulation operations in each arithmetic cycle on data rows (e.g., input feature rows) selected from the first buffer circuit and data rows (e.g., weight rows) selected from the second buffer circuit, respectively. For simplicity, the following description refers to the processing within a single slave processing circuit SL; it is understood that similar processing is performed in other SLs.

[0137] As described above, a single slave processing circuit may need to process one or more convolutional kernels with Co values. In some embodiments, the operation can be performed in a multi-layered loop in dimensional order, where the Co value assigned to each slave processing circuit is the innermost loop, with the number of loops Nco = Aco / Ns; the Kw dimension of the convolutional kernel is the middle loop, with the number of loops Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolutional kernel, and Kmax is the maximum convolutional kernel width supported by the slave processing circuit, which is related to the size of the first buffer circuit, the width dimension folding factor Ws, etc.; the Kh dimension and Ci dimension of the convolutional kernel are divided into blocks Bci according to Pci, with the number of loops Nkh = Kh, Nci = Bci = ceil(Ci / Pci), where Kh is the height dimension of the convolutional kernel; thus, the total number of loops Ncycle = Nco * Nkw * Nkh * Nci.

[0138] As can be seen from the innermost loop, when multiple Co values ​​need to be processed, the input feature map can be reused. The number of times the input feature map is reused is rn = Pco / Ns. For example, when Pco = 64 and Ns = 16, each slave processing circuit processes 4 Co values, and the input feature map is reused rn = 64 / 16 = 4 times in each slave processing circuit.

[0139] In the innermost loop, for each Co value, the output feature map can be computed in parallel by multiple computational circuits (CUs) within a single processing circuit. Considering the dimensional storage order of the output feature map and the W-folding of the input feature map, to simplify output processing, preferably, Ncu output blocks are sequentially divided along the Wo dimension, so that Ncu computational circuits can perform parallel computations on each block, with each output block corresponding to the computation result of one input feature data line. In some embodiments, Ncu adjacent input feature data lines are sequentially selected from the first buffer circuit and distributed to the Ncu computational circuits, and a corresponding extended weight data line is selected from the second buffer circuit and broadcast to the Ncu computational circuits, thereby achieving parallel computation of the Ncu output blocks by reusing the weight data.

[0140] Figure 9A schematic splitting method of the output feature map according to an embodiment of this disclosure is shown. For simplicity, Figure 9 This example only shows the splitting of the output feature map of one Co value in the innermost loop along the Wo dimension. In this example, assuming Ncu = 4, the output is divided into 4 blocks sequentially along the Wo dimension, with each block corresponding to the result of the operation on one row of input feature data.

[0141] Furthermore, depending on the different data formats within a data row of the input feature map, the output block computed by a single computational circuit CU can include different numbers of output points. Specifically, based on the previously determined width dimension folding factor Ws, each output block includes Ws consecutive output points along the width Wo dimension. For example, when the input feature map Ci is split at a granularity of Pci = 16B, a data row includes 4 Wi, and output points in 4 Wo dimensions can be computed; when the input feature map Ci is split at a granularity of Pci = 32B, a data row includes 2 Wi, and output points in 2 Wo dimensions can be computed; and when the input feature map Ci is split at a granularity of Pci = 64B, a data row includes 1 Wi, and output points in 1 Wo dimension can be computed. Figure 9 The diagram further illustrates the different configurations of a single output block in the three cases described above, including 4, 2, or 1 Wo output points, respectively.

[0142] To support a single CU simultaneously computing one or more Wo output points that may be included in an output block, in some embodiments, the corresponding weight data can be constructed as follows: When distributing the convolution kernel in the second storage circuit to the second buffer circuit of each slave processing circuit, in addition to aligning the Ci dimension of the convolution kernel to Pci, a column of Ci data split or aligned to Pci in the Ci dimension is copied and expanded into Ws columns according to the folding factor Ws in the width dimension, forming an extended weight data row, which is stored in the second buffer circuit. That is, the shape of an extended weight data row is Ws*Pci, which can correspond to an input feature data row. Thus, an extended weight data row can be selected from the second buffer circuit and broadcast to N in the slave processing circuit. CU Each arithmetic circuit can then perform bitwise multiplication and accumulation on a data line of Pci / M = 1 / Ws, for an input feature line from the first buffer circuit and an extended weight data line from the second buffer circuit, to obtain a partial sum of M / Pci = Ws output points.

[0143] Therefore, the above calculation process involves two layers of weight reuse: the first layer is between the computational circuits (CUs), where weights are broadcast to Ncu computational circuits, resulting in Ncu reuse times; the second layer is between one or more Wo output points within each computational circuit, where weights are expanded to calculate M / Pci output points within each CU, resulting in M / Pci reuse times. Thus, by reusing data as much as possible, frequent data access and memory access can be effectively reduced.

[0144] In the Kw-dimensional intermediate loop, the input feature data rows and expanded weight data rows are synchronously selected by sliding in the width dimension with a step size of 1 on the first and second buffer circuits. As mentioned earlier, the number of sliding steps in the intermediate loop, i.e., the number of loops, is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel, and Kmax is the maximum convolution kernel width value supported by the processing circuit.

[0145] In some embodiments, Kmax can be determined as follows:

[0146] Kmax = L1*Ws - Ncu*Ws + 1

[0147] Where L1 is the size of the first buffer circuit, in data rows; Ncu is the number of scheduled computational circuits; and Ws is the folding factor of the width dimension. For example, with a first buffer circuit of L1 = 8 data rows and Ncu = 4, Kmax = 17 when Ws = 4; Kmax = 9 when Ws = 2; and Kmax = 5 when Ws = 1. It can be seen that in most cases, the width dimension Kw of the convolution kernel will not exceed Kmax; therefore, Nkw = Kw.

[0148] The outer loop can include loops for the Kh and Ci dimensions, each split into blocks (Bci) according to Pci. The loop order for the two dimensions can be interchanged. In one example, the Kh dimension loop can be performed first, followed by the Bci loop. In another example, the Bci loop can be performed first, followed by the Kh dimension loop. Since the Kh dimension data is not folded or split in any way, the convolution scheme of this disclosure embodiment can support convolution strides of arbitrary values ​​in the Kh dimension. Setting the Bci-level loop in the outer loop is beneficial for fully reusing the loaded data in the inner loops without frequently updating the buffer circuitry.

[0149] It's understandable that when the width of the convolutional kernel exceeds Kmax, it needs to be split along the Kw direction according to that maximum kernel width. In this case, in addition to the three loops mentioned above, further loop processing is performed according to the split along Kw.

[0150] It can also be understood that when the size of the Wo dimension of the output feature map exceeds the computational cost in a single operation, for example, Wo > Ncu * Ws, then Wo can be processed cyclically by splitting it according to Ncu * Ws. Similarly, Ho and Co can also be processed cyclically as needed, that is, they can be processed sequentially in the order of Wo -> Ho -> Co, where Co is split into 64.

[0151] When the processing circuit writes the calculation results from the arithmetic circuit, it can store the calculation results of each arithmetic circuit in, for example, the order of Wo dimension first, then Co dimension. Figure 5 In the third buffer circuit, when the processing circuit outputs the output points of its internal arithmetic circuits, it can output the output points calculated by multiple arithmetic circuits in a specific order according to the division of output points, so that the continuously output points are continuous in the Co dimension, which facilitates subsequent processing. For example, when the processing circuit outputs the output points of each arithmetic circuit, it can output them in the order of Co dimension first, then Wo dimension.

[0152] As can be seen from the preceding calculation process, each time Ncu*Ws output points are calculated in the Wo dimension of the output feature map. That is, the output points are aligned to Ncu*Ws. Therefore, there may be redundant output points calculated. These redundant output points in the Wo dimension can be filtered out in the data path storing the calculation results.

[0153] Furthermore, on the data path storing the computation results, redundant output points in the Co dimension due to alignment can be filtered out. In some examples, the main processing circuit can control the execution of this filtering operation. For instance, the main processing circuit can read the output results from the third buffer circuit of each slave processing circuit, filter out redundant output points, and then write them to the first storage circuit. In other examples, the slave processing circuit can also control the output results from the third buffer circuit, filter out redundant output points, and then transmit them to the first storage circuit for storage on the data path.

[0154] Optionally or additionally, different data read / write schemes can be used for the operation results during the transfer from the third buffer circuit to the first storage circuit. These read / write schemes can, for example, refer to the read / write schemes for the input feature map described above. The following is only a brief description; a detailed description can be found above. In some embodiments, the writing scheme for the operation results can be implemented in the last step of writing to the first storage circuit, that is, it can be performed on the data path storing the operation results, after dimensional transformation and data filtering of the operation results. Those skilled in the art will understand that the order of these processes can also be adjusted, and the embodiments disclosed herein are not limited thereto.

[0155] In some embodiments, to improve transmission and write efficiency, the output channel Co dimension of the output feature map can be split according to the output channel splitting granularity Pco, and the data of each splitting granularity can be combined with the data of the width dimension Wso times to form the output feature data row for transmission, where Pco*Wso=M. The output channel splitting granularity Pco can be determined similarly to the input channel splitting granularity Pci mentioned above, and will not be detailed here. Thus, similarly, in these embodiments, depending on the number of split blocks on the Co dimension, the operation results can be saved to the storage circuit by continuous writing or interval (non-contiguous) writing. Note that "contiguous" or "interval" in this article refers to the data being continuous or intermittent in the storage address.

[0156] In one example, when the number of partitioned blocks in dimension Co is 1, i.e., when dimension Co is not partitioned, Co ≤ Pco, and a W contains only one partitioned block of size Co. When storing the operation result, output feature map data of size Co*Wso can be read from the operation result of the third buffer circuit. At this time, when these Wso Ws are stored, these Ws are contiguous. Therefore, in this case, the read output feature map data can be written continuously into the first storage circuit. This can be done by writing Wso Ws of data at once with a single write request, thus simplifying the instructions. Of course, multiple write requests can also be used to write contiguous data of Wso Ws.

[0157] In another example, when the number of partitions along the Co dimension is not 1, i.e., when the Co dimension is partitioned into multiple partition granularities Pco, Co > Pci. Since a data row consists of Wso * Pco, it is necessary to write data of size Pco across each W at intervals. Furthermore, considering that Co may not be divisible by Pco, the Co dimension can be divided into an integer part and a remainder part according to Pco. Subsequently, the integer part and the remainder part can be processed separately.

[0158] In one implementation, for the integer part, output feature map data of size Pco*Wso is read from the output calculation result each time. This output feature map data can be divided into Wso output feature map data blocks, each containing data of size Pco in the Co dimension. It can be understood that, considering the final desired data storage dimension is HoWoCo, the data within these output feature map data blocks needs to be stored contiguously, while the output feature map data blocks need to be stored at intervals of 1*Co(WC), or 1 W. Therefore, in this case, the read Wso output feature map data blocks can be written to the first storage circuit at intervals, with the starting address of each written output feature map data block sequentially spaced Co apart. That is, the amount of data to be written each time (Pco) requires Wso writes. In other words, Wso writes are performed, each time writing the amount of data of Pco on a W-sized scale at equal intervals. This can be achieved through Wso write requests, with each write request writing the corresponding amount of data of Pco on a W-sized scale.

[0159] In another implementation, for the remainder portion, output feature map data of size Co_rem*Wso is read from the output calculation result each time. This output feature map data can also be divided into Wso output feature map data blocks. Each output feature map data block includes data of size Co_rem in the Co dimension, where Co_rem represents the remainder obtained by dividing Co by Pco. It can be understood that the remainder portion is usually at the end of the integer portion. Similarly, the read Wso output feature map data blocks can be written into the first storage circuit at intervals, with the starting address of each written output feature map data block sequentially spaced Co. That is, each time the amount of data Co_rem is written, it needs to be written Wso times. In other words, Wso times, the amount of data Co_rem on W is written at equal intervals each time. At this time, data writing can be achieved through Wso write requests, with each write request writing the amount of data Pco on the corresponding W. It can be seen that the interval writing processing of the remainder portion is similar to that of the integer portion, only the amount of data written is slightly different.

[0160] By using the above-mentioned interval writing method, the write bandwidth can be fully utilized each time a write operation is performed, and the written data can maintain a continuous storage order in the Co dimension on the storage circuit.

[0161] In the above embodiments, each slave processing circuit can control the reading method of the weight data row and the input feature map data row according to the weight reuse and / or input feature map reuse method, so as to perform bitwise multiplication and accumulation operations on the entire convolution window of the convolution output point simultaneously through multiple operations to obtain multiple partial sums and results, and accumulate them to obtain the convolution output at the corresponding convolution output point.

[0162] The following describes the detailed operation process of convolution in different scenarios of the embodiments disclosed herein, with reference to several specific examples.

[0163] Example 1: Ci aligned to 16B, number of split blocks Bci = 1, Co aligned to 64B.

[0164] In this embodiment, Ci is aligned to 16B, so a row of input feature data is formatted as 4×16B (WiCi), resulting in an output block of 1×4 (CoWo) output points computed by a single computational circuit. Co is aligned to 64, so when Ns = 16 slave processing circuits are schedulable, each slave processing circuit processes 4 Co values. Let's assume the convolution kernel size is KhKw = 3×3.

[0165] Figures 10a-10d A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 1 of this disclosure is shown. In this example, the size of the first buffer circuit is 8×64B, meaning it can buffer up to 8 rows of data. The diagram shows that each row of data includes 4 columns of Wi data. The size of the second buffer circuit is 9×64B, meaning it can buffer up to 9 rows of data.

[0166] Figure 10a The diagram illustrates the operation process of the innermost Co loop and input feature map multiplexing within a loop where Kh=0 and Kw=0. N is selected from the first buffer circuit according to the method corresponding to the output block partitioning. CU Each of the N input feature lines is sent to N. CU A computational circuit is used for calculation. During the subsequent Aco / Ns = 4 calculations, this N CU The input feature rows are respectively in N CU Each operational circuit is reused 4 times.

[0167] During these four calculations, weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to N in the processing circuit. CU There are several computational circuits, where each weight row is formed by expanding 1 / 4 of a weight row by copying it three times. For example, in the example shown in the figure, suppose the processing circuit is SL0, which processes four convolutional kernels with Co = 0, 16, 32, and 48.

[0168] During the first calculation period indicated by arrow ①, a 16B*4 copy of the 16B (Ci dimension) data stored in the 0th row of the second buffer circuit, corresponding to Kw=0 in the convolution kernel where Co=0, is broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row at this time includes four Wi values ​​and the weights are copied four times, bitwise multiplication and accumulation are performed according to 1 / 4 of the data row, thus obtaining four output points where Co=0. Specifically, the operation circuit CU0 calculates output points w0, w1, w2, and w3, the operation circuit CU1 calculates output points w4, w5, w6, and w7, and so on. It can be understood that the output points of these calculations are partial sums, only accumulating the data in the Ci direction when Kh=0 and Kw=0.

[0169] During the second calculation period indicated by arrow ②, 16B*4 copies of the 16B (Ci dimension) data corresponding to Kw=0 in the convolution kernel of Co=16, stored in the first row of the second buffer circuit, are broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row at this time includes 4 Wi and the weights are copied 4 times, bitwise multiplication and accumulation are performed according to 1 / 4 of the data row, thus obtaining 4 output points of Co=16. Specifically, the operation circuit CU0 calculates output points w0, w1, w2, and w3, the operation circuit CU1 calculates output points w4, w5, w6, and w7, and so on.

[0170] Similarly, during the third calculation period indicated by arrow ③, the four CUs calculate the output points in the 16 Wo directions for Co = 32; during the fourth calculation period indicated by arrow ④, the four CUs calculate the output points in the 16 Wo directions for Co = 48, which will not be elaborated here. Thus, for the four Co values ​​processed by a single slave processing circuit, the processing is cyclically performed to obtain the output points in the 16 Wo dimensions for each of the four Co values. In the operation of each CU, the input feature row is multiplexed four times, while in the operation of a single slave processing circuit, the weights are multiplexed Ncu*M / Pci = 4*(64 / 16) = 16 times.

[0171] After the innermost Co loop is completed, the intermediate Kw loop can then be performed.

[0172] Figure 10b The loop processing for Kh=0 and Kw=1 is shown. Then, for Kw=1, the same process is performed. Figure 10aThe computational process involves a similar Co loop and input feature map reuse. The difference lies in the selection of data for computation: Four corresponding input feature rows are selected by sliding one step in the Wi direction from the first buffer circuit and sent to the four computation circuits respectively; and weight data corresponding to different Co values, Kh=0 and Kw=1, are sequentially selected from the second buffer circuit. During the subsequent four calculations, these four input feature rows are reused four times in each of the four computation circuits, and each time they are processed with extended weight rows of different Co values.

[0173] For example, during the first calculation as shown by arrow ①, 16B*4 copies of the 16B (Ci dimension) data stored in the fourth row of the second buffer circuit, corresponding to the convolution kernel with Co=0 and Kw=1, are broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. It can be understood that the output points of these calculations are partial sums, accumulating only the data in the Ci direction when Kh=0 and Kw=1. It can also be understood that since the input feature row has slid one step in the Wi direction, the weight Kw has also slid one step in the W direction (Kw=1). Therefore, these partial sums still belong to the 16 output points w0~w15 in the Wo dimension with Co=0. Therefore, the partial sums obtained in this calculation are accumulated in the previous ( Figure 10a The partial sum obtained from the operation in )

[0174] Similarly, during the second calculation period indicated by arrow ②, the four CUs calculate the partial sum of the output points in the 16 Wo directions with Co = 16, and accumulate them. Figure 10a The partial sums obtained during the corresponding calculation cycle are calculated. In the third calculation period indicated by arrow ③, the four CUs calculate and accumulate the partial sums of the output points in the 16 Wo directions with Co = 32; in the fourth calculation period indicated by arrow ④, the four CUs calculate and accumulate the partial sums of the output points in the 16 Wo directions with Co = 48. This will not be elaborated further here.

[0175] Therefore, Kw in the intermediate layer enters the next cycle, that is, Kw = 2.

[0176] Figure 10c The loop processing for Kh=0 and Kw=2 is shown. Then, for Kw=2, the same process is performed. Figure 10a and Figure 10bThe operation process involves a similar Co loop and input feature map reuse. The difference lies in the selection of data for the operation: In this case, the first buffer circuit slides one step further in the Wi direction to select the corresponding four input feature rows, which are then sent to the four operation circuits respectively; and the second buffer circuit sequentially selects weight data corresponding to different Co values, Kh=0 and Kw=2. As shown in the diagram, due to the limited buffer space of the second buffer circuit, data is loaded cyclically. Data with Kw=2 is placed in rows 0-3, flushing out the original weight data with Kw=0. It is understood that the size of the buffer circuit is not limited to the illustrated example, and the corresponding storage order is not limited to the illustrated scenario. Those skilled in the art can adapt and use it according to the actual hardware being used.

[0177] The parts and results of the four calculation cycles shown in the diagram are added to the previous accumulated result. The calculation process is similar to that before and will not be described again here.

[0178] Therefore, after the intermediate Kw loop completes its processing, the outer Kh loop can proceed. In this example, since the number of split blocks in the Ci dimension is 1, there is no need to perform a loop on the split blocks in the Ci dimension.

[0179] Figure 10d The loop processing for Kh=1 and Kw=0 is shown. At this point, for Kh=1, the same process is performed. Figure 10a The operation process is similar to Co loop and input feature map reuse. The difference lies in the selection of data to participate in the operation: at this time, the input feature data in the first buffer circuit is updated to the feature row corresponding to Hi=1, and the corresponding 4 input feature rows are selected from the first row and sent to the 4 operation circuits respectively; and the weight data corresponding to different Co values ​​Kh=1 and Kw=0 are selected sequentially from the second buffer circuit.

[0180] For example, during the first calculation period indicated by arrow ①, 16B*4 copies of the 16B (Ci dimension) data stored in the fourth row of the second buffer circuit, corresponding to the convolution kernel with Co=0 and Kh=1 and Kw=0, are broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. It can be understood that the output points of these calculations are partial sums, accumulating only the data in the Ci direction when Kh=1 and Kw=0. It can also be understood that since Hi=1 and the weight Kh=1 of the input feature row, these partial sums still belong to the 16 output points w0~w15 in the Wo dimension with Co=0. Therefore, the partial sums obtained in this calculation are accumulated in the previous ( Figure 10c The partial sum obtained from the operation is shown above. The other three calculation cycles are similar and will not be repeated.

[0181] Therefore, after the outer Kh loop has finished processing, each arithmetic circuit can accumulate to obtain the final convolution result of 4 Cos, each with 4 output points. One slave processing circuit with 4 arithmetic circuits obtains 4 Cos, each with 4*4 = 16 output points. The 16 slave processing circuits obtain a total of 16*4 = 64 Cos, each with 16 output points.

[0182] It is understandable that, limited by the sizes of the first and second buffer circuits, when Kw exceeds a certain value, it is necessary to traverse along the Kw direction while switching inputs and weights until a complete output is calculated. Furthermore, when the size of the output feature map exceeds the computational complexity of a single operation, it can be processed sequentially in the order Wo->Ho->Co, where Co is split into 64 parts. Those skilled in the art can similarly deduce the calculation process based on the foregoing description, and it will not be elaborated further here.

[0183] Figure 11 A schematic diagram of the writing and output logic of the calculation results according to Embodiment 1 of this disclosure is shown.

[0184] As shown in the figure, multiple arithmetic circuits (CUs) within a single slave processing circuit (SL) can sequentially write the computation results into a result buffer circuit (e.g., according to the computation order). Figure 5 In the third buffer circuit. Specifically, the output points of the same Co calculated by each CU can be written first in the order of Wo (writing loop ①). Then, the output points of different Co calculated by each CU can be written in the order of Co (writing loop ②). For example, for SL0, first write w0~w15 when Co=0, then write w0~w15 when Co=16, then w0~w15 when Co=32, and finally w0~w15 when Co=48. Similar result writing is performed in other SLs, except that the Co values ​​are processed differently.

[0185] As can be seen from the above writing order, the operation results in the result buffer circuit are stored in CW (W in the lowest storage dimension) order. However, the final output result is expected to be in HWC storage order (C in the lowest storage dimension). Therefore, in the data path for reading the result from the result buffer circuit, the data can be converted from CW order to WC order.

[0186] Specifically, as shown in the figure, the first output point w0 of the W dimension can be read from the result buffer circuit of each processing circuit in the order of Co (reading loop ①), and then the output points on each Wo can be read in the order of Wo (reading loop ②). Figure 11 The right-hand view shows the readout results. Note that when reading in Co order, the results are read in turn on the 16 SL result buffer circuits so that the Co dimensions are continuous, for example from 0 to 63.

[0187] As can be seen from the above calculation process, since the adjacent W dimension is used to compensate for the lack of Ci dimension, there is no need to perform additional slewing processing on the input feature map and convolution kernel in software, simplifying the processing and reducing instruction overhead. Furthermore, the output data format also shows that since the data is converted from CW order to WC order in the read data path, there is no need to perform additional slewing processing on the output feature map in software, avoiding complex software parameter configuration. In addition, the loop process shows that since the output feature map is split only in the Wo dimension, it can support convolution strides of arbitrary size in the H dimension.

[0188] Example 2: Ci aligned to 32B, number of split blocks Bci = 1, Co aligned to 32B.

[0189] In this embodiment, Ci is aligned to 32B, so the format of an input feature data line is 2×32B (WiCi), and thus the output block calculated by one arithmetic circuit includes 1×2 (CoWo) output points. Co is aligned to 32, so when Ns = 16 slave processing circuits are schedulable, each slave processing circuit processes 2 Co values. Let's assume the size of the convolution kernel is KhKw = 3×3.

[0190] Figure 12 A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 2 of this disclosure is shown. In this example, the first buffer circuit and the second buffer circuit are configured the same as in Embodiment 1.

[0191] The diagram illustrates the operation process of the innermost Co loop and input feature map multiplexing within a loop where Kh = 0 and Kw = 0. Following the method corresponding to the output block partitioning, four input feature lines are selected from the first buffer circuit and sent to four arithmetic circuits for calculation. During the subsequent Aco / Ns = 2 calculations, these four input feature lines are multiplexed twice in each of the four arithmetic circuits.

[0192] During these two calculations, weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the four operational circuits within the slave processing circuit. Each weight row is formed by expanding a copy of half a weight row. For example, in the example shown in the figure, assume the slave processing circuit is SL0, which processes two convolutional kernels with Co = 0 and 16.

[0193] During the first calculation period indicated by arrow ①, a 32B*2 copy of the 32B (Ci dimension) data stored in the 0th row of the second buffer circuit, corresponding to Kw=0 in the convolution kernel when Co=0, is broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row at this time includes 2 Wi and the weights are copied twice, bitwise multiplication and accumulation are performed according to half a data row, thus obtaining two output points when Co=0. Specifically, the operation circuit CU0 calculates output points w0 and w1, the operation circuit CU1 calculates output points w2 and w3, and so on. It can be understood that the output points of these calculations are partial sums, only accumulating the data in the Ci direction when Kh=0 and Kw=0.

[0194] Similarly, during the second calculation period indicated by arrow ②, the four CUs calculate the output points in the eight Wo directions for Co = 16, which will not be elaborated here. Thus, for the two Co values ​​processed by a single slave processing circuit, the processing is cyclically repeated to obtain the output points in the eight Wo dimensions for each of the two Co values. In the operation of each CU, the input feature row is multiplexed twice, while in the operation of a single slave processing circuit, the weights are multiplexed Ncu*M / Pci = 4*(64 / 32) = 8 times.

[0195] After the innermost Co cycle is completed, the middle Kw cycle can be performed, followed by the outer Kh cycle. The cycle process can be referred to in Example 1, and will not be repeated here.

[0196] Example 3: Ci aligned to 32B, number of split blocks Bci = 1, Co aligned to 16

[0197] In this embodiment, Ci is aligned to 32B, so the format of an input feature data line is 2×32B (WiCi), and thus the output block computed by one arithmetic circuit includes 1×2 (CoWo) output points. Co is aligned to 16, so when Ns = 16 slave processing circuits are schedulable, each slave processing circuit processes one Co value. Let's assume the size of the convolution kernel is KhKw = 3×3.

[0198] Figure 13 A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 3 of this disclosure is shown. In this example, the first buffer circuit and the second buffer circuit are configured the same as in Embodiments 1 and 2.

[0199] The diagram illustrates the operation process of the innermost Co loop and input feature map multiplexing within a loop where Kh=0 and Kw=0. Following the method corresponding to the output block division, four input feature lines are selected from the first buffer circuit and sent to four arithmetic circuits for calculation. During the subsequent Aco / Ns=1 calculation, these four input feature lines are used once in each of the four arithmetic circuits; that is, there is no multiplexing.

[0200] For the example in the figure, assume the processing circuit is SL0, which processes a convolution kernel with Co = 0.

[0201] During the first calculation (indicated by arrow ①), a 32B*2 copy of the 32B (Ci dimension) data stored in the 0th row of the second buffer circuit, corresponding to Kw=0 in the convolution kernel where Co=0, is broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row includes two Wi values ​​and the weights are copied twice, bitwise multiplication and accumulation are performed on half a data row, resulting in two output points where Co=0. Specifically, the operation circuit CU0 calculates output points w0 and w1, the operation circuit CU1 calculates output points w2 and w3, and so on. It can be understood that these calculated output points are partial sums, accumulating only the data in the Ci direction when Kh=0 and Kw=0.

[0202] Therefore, for a single Co value processed by a single slave processing circuit, the output points of the eight Wo dimensions of this Co can be obtained through a single calculation. In the operation of each CU, the input feature row is used once, while in the operation of a single slave processing circuit, the weights are reused Ncu*M / Pci4*(64 / 32) = 8 times.

[0203] After the innermost Co cycle is completed, the middle Kw cycle can be performed, followed by the outer Kh cycle. The cycle process can be referred to in Example 1, and will not be repeated here.

[0204] Example 4: Ci aligned to 64B, number of split blocks Bci = 1, Co aligned to 32B.

[0205] In this embodiment, Ci is aligned to 64B, so a row of input feature data is formatted as 1×64B (WiCi), and the output block computed by one arithmetic circuit comprises 1×1 (CoWo) output points. Co is aligned to 32, so when Ns = 16 slave processing circuits are schedulable, each slave processing circuit processes 2 Co values. Let's assume the convolution kernel size is KhKw = 3×3.

[0206] Figure 14 A schematic diagram of the operation process of the convolution operation scheme according to Embodiment 4 of this disclosure is shown. In this example, the first buffer circuit and the second buffer circuit are configured the same as in Embodiments 1, 2 and 3.

[0207] The diagram illustrates the operation process of the innermost Co loop and input feature map multiplexing within a loop where Kh = 0 and Kw = 0. Following the method corresponding to the output block partitioning, four input feature lines are selected from the first buffer circuit and sent to four arithmetic circuits for calculation. During the subsequent Aco / Ns = 2 calculations, these four input feature lines are multiplexed twice in each of the four arithmetic circuits.

[0208] During these two calculations, weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the four operational circuits within the slave processing circuit. Each weight row directly corresponds to the original weight row, without the need for copying or expansion. For example, in the example shown in the figure, suppose the slave processing circuit is SL0, which processes two convolutional kernels with Co = 0 and 16.

[0209] During the first calculation period indicated by arrow ①, the 64B (Ci dimension) data stored in the 0th row of the second buffer circuit, corresponding to Kw=0 in the convolution kernel when Co=0, is broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row at this time includes one Wi and only one weight, bitwise multiplication and accumulation are performed on a single data row, resulting in one output point when Co=0. Specifically, the operation circuit CU0 calculates output point w0, the operation circuit CU1 calculates output point w1, and so on. It can be understood that the output points of these calculations are partial sums, only accumulating the data in the Ci direction when Kh=0 and Kw=0.

[0210] Similarly, during the second calculation period indicated by arrow ②, the four CUs calculate the output points in the four Wo directions for Co = 16, which will not be elaborated here. Thus, for the two Co values ​​processed by a single slave processing circuit, the processing is cyclically repeated to obtain the output points in the four Wo dimensions for each of the two Co values. In the operation of each CU, the input feature row is multiplexed twice, while in the operation of a single slave processing circuit, the weights are multiplexed Ncu*M / Pci = 4*(64 / 64) = 4 times.

[0211] After the innermost Co cycle is completed, the middle Kw cycle can be performed, followed by the outer Kh cycle. The cycle process can be referred to in Example 1, and will not be repeated here.

[0212] Example 5: Ci aligned to 16B, number of split blocks Bci = 3, Co aligned to 32B.

[0213] Unlike the previous embodiments, in this embodiment, there are multiple split blocks along the Ci dimension. Specifically, Ci = 48B, the splitting granularity Pci = 16B, resulting in 3 split blocks. Therefore, in the loop calculation, the outer loop has an additional loop at the split block level.

[0214] At this point, an input feature data row is formatted as 4×16B (WiCi), so the output block calculated by one computational circuit includes 1×4 (CoWo) output points. Co is aligned to 32, so when Ns = 16 slave processing circuits are schedulable, each slave processing circuit processes 2 Co values. Let's assume the size of the convolution kernel is KhKw = 3×3.

[0215] Figure 15 A schematic diagram of the operation process of the convolution operation scheme according to Embodiment 5 of this disclosure is shown. In this example, the first buffer circuit and the second buffer circuit are configured the same as in Embodiments 1, 2, 3 and 4.

[0216] The diagram illustrates the operation process of the innermost Co loop and input feature map multiplexing within a loop where Bci=0, Kh=0, and Kw=0. Following the division method corresponding to the output block, four input feature lines are selected from the first buffer circuit and sent to four arithmetic circuits for calculation. These input feature lines correspond to the data portions of the input feature map where Hi=0, Bci=0, and Wi=0~15. During the subsequent Aco / Ns=2 calculations, these four input feature lines are multiplexed twice in each of the four arithmetic circuits.

[0217] During these two calculations, weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the four computational circuits within the processing circuit. Each weight row is formed by expanding a quarter weight row by copying it three times. This quarter weight row is obtained by splitting it in the Ci dimension with a granularity of Pci = 16B. For example, in the example in the figure, assuming the processing circuit is SL0, it processes two convolutional kernels with Co = 0 and 16, and takes the data portion of Kw = 0, Kh = 0, and Bci = 0 from each convolutional kernel for copying and expansion.

[0218] During the first calculation period indicated by arrow ①, a 16B*4 copy of the 16B (Ci dimension) data corresponding to Kw=0 in the convolution kernel of Co=0, stored in the 0th row of the second buffer circuit, is broadcast to the four CUs for bitwise multiplication and accumulation with their respective received input feature rows. Note that since each input feature row at this time includes 4 Wi and the weights are copied 4 times, bitwise multiplication and accumulation are performed according to 1 / 4 of the data row, thus obtaining one output point of Co=0. Specifically, the operation circuit CU0 calculates the output point w0, the operation circuit CU1 calculates the output point w1, and so on. It can be understood that the output points of these calculations are partial sums, only accumulating the data of Kh=0, Kw=0, and Bci=0.

[0219] Similarly, during the second calculation period indicated by arrow ②, the four CUs calculate the output points in the 16 Wo directions for Co = 16, which will not be elaborated here. Thus, for the two Co values ​​processed by a single slave processing circuit, the processing is cyclically repeated to obtain the output points in the 16 Wo dimensions for each of the two Co values. In the operation of each CU, the input feature row is multiplexed twice, while in the operation of a single slave processing circuit, the weights are multiplexed Ncu*M / Pci = 4*(64 / 16) = 16 times.

[0220] After the innermost Co cycle is completed, the middle Kw cycle can be performed, followed by the outer Kh and Bci cycles. The Kw and Kh cycle processes can be referred to in Example 1, and will not be repeated here.

[0221] For the Bci loop, in one implementation, the data in the first and second buffer circuits can be updated after the Kw loop. For example, after the calculation for the data block with Bci=0 is completed, the data block with Bci=1 is loaded, and the loop is performed again for the Kw and Co dimensions. The partial sums obtained in each calculation are still the corresponding output points, so they can be accumulated onto the previous partial sum results. In another implementation, the data in the first and second buffer circuits can be updated after the Kh loop. The calculation process is similar to the previous one and will not be detailed here.

[0222] Example 6: Example of non-contiguous writing of calculation results

[0223] In this embodiment, the processing performed on the data path from the third buffer circuit to the first storage circuit is described. In this example, it is assumed that the original output channel Co = 47, which is aligned to 48 during the operation, and the output Wo = 16.

[0224] Figure 16 This diagram illustrates the logic for writing and outputting computation results according to an embodiment of this disclosure.

[0225] As shown in the figure and the preceding description, multiple arithmetic circuits (CUs) within a single slave processing circuit (SL) can sequentially write the computation results into the result buffer circuit (e.g., ...) according to the computation order. Figure 5 In the third buffer circuit. Specifically, the output points of the same Co calculated by each CU can be written first in the order of Wo (writing loop ①). Then, the output points of different Co calculated by each CU can be written in the order of Co (writing loop ②).

[0226] On the data path from which the result is read from the result buffer circuit, the data can be converted from CW order to WC order.

[0227] In addition, output points in the Co dimension that are redundantly calculated due to alignment can be filtered out. For example, the calculation of SL15 for Co=47 is a result of redundant calculation for alignment and can be filtered out.

[0228] Furthermore, as mentioned earlier, in order to improve transmission and writing efficiency, the output channel Co dimension of the output feature map can be split according to the output channel splitting granularity Pco, and the data of each splitting granularity can be combined with the data of the width dimension Wso times to form the output feature data row for transmission, where Pco*Wso=M.

[0229] In this example, since Co = 47B (assuming int8 data type), we can choose Pco = 16B for splitting, thus maximizing the use of transmission bandwidth. Based on Pco, the number of blocks to be split along the Co dimension is determined to be 3. Furthermore, since Co is not divisible by Pco, it can be divided into an integer part (Co = 0~31) and a remainder part (Co = 32~46).

[0230] For the integer part, as shown in the figure, output feature map data of size Pco*Wso = 16B*4 can be read from the calculation result each time. That is, it is read from the result buffer circuits of the 16 SLs in turn according to the Co order. Specifically, the four output points w0, w1, w2 and w3 in the Wo direction when Co = 0 are read from SL0 (ellipse 1601); the four output points w0, w1, w2 and w3 in the Wo direction when Co = 1 are read from SL1 (ellipse 1602); until the four output points w0, w1, w2 and w3 in the Wo direction when Co = 15 are read from SL15 (not shown).

[0231] The read output feature map data can be divided into Wso output feature map data blocks, each containing 16 bytes of data in the Co dimension. Specifically, as shown in the figure, the read 16*4 output feature map data can be divided into four output feature map data blocks (1611~1614) according to the Wo dimension, each containing 16 bytes of data in the Co dimension. To obtain the desired data storage dimension HoWoCo, the data within these output feature map data blocks needs to be stored contiguously, while the output feature map data blocks need to be stored intermittently, with an interval of 1*Co(WC), or 1 W, which is 47 bytes in this example. Therefore, in this case, the four read output feature map data blocks can be written intermittently into the first storage circuit, with the starting address of each written output feature map data block sequentially spaced Co. That is, each 16 bytes of data needs to be written four times.

[0232] The second row of data in the integer part can be processed similarly.

[0233] For the remainder portion, i.e., the data portion where Co = 32 to 46, output feature map data of size Co_rem * Wso is read from the calculation result. It is understood that redundant calculations in the Co dimension due to alignment can be skipped during reading from the third buffer circuit, or discarded after reading; this disclosed embodiment is not limited in this respect. In this example, Co_rem = 15B. This output feature map data of the remainder portion can also be divided into Wso output feature map data blocks, each including data of size Co_rem in the Co dimension. As shown in the figure, the remainder portion ends at the end of the integer portion. Similarly, the read Wso output feature map data blocks can be written to the first storage circuit at intervals, with the starting address of each written output feature map data block sequentially spaced Co. In this example, 15B of data is written each time, and the writing is performed 4 times.

[0234] Next, similar processing can be performed on the remaining data in the Wo dimension, which will not be repeated here.

[0235] The above description and explanation of the convolution optimization scheme provided in this disclosure, combined with the specific convolution operation processes of six embodiments, provide an exemplary description and explanation. It is understood that, depending on the different values ​​of Ci and Co, there can be many more combinations to obtain different embodiments. Furthermore, based on the teachings of this disclosure, those skilled in the art can conceive of other convolution optimization schemes according to specific hardware circuit configurations (such as the number of processing circuits, the number of operational circuits within the processing circuits, and the hardware's single-processing capability), all of which fall within the scope of this disclosure and will not be listed here.

[0236] This disclosure also provides a method for performing convolution operations using the aforementioned computing device. Those skilled in the art will understand that the steps of performing convolution operations correspond to the various circuits of the computing device described above in conjunction with the accompanying drawings; therefore, the features described above also apply to the method steps and will not be repeated here.

[0237] This disclosure also provides a chip that may include the computing device of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

[0238] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0239] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0240] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0241] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0242] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0243] The foregoing can be better understood in accordance with the following terms:

[0244] Clause 1. A computing device configured to perform a convolution operation, the computing device comprising:

[0245] A storage circuit that stores input feature maps and convolutional kernels, wherein the convolutional kernels are stored in blocks according to the output channel Co dimension for allocation to different slave processing circuits;

[0246] The main processing circuit aligns the input channel Ci dimension of the input feature map to the input channel splitting granularity Pci, and combines the data of each splitting granularity with the data of the width dimension Ws times to form an input feature data row, which is then broadcast to multiple slave processing circuits to perform the convolution operation, where Pci*Ws = M, and M is the amount of data processed in a single hardware operation; and

[0247] The plurality of said slave processing circuits are configured to perform convolution operations on the broadcast input feature map and the corresponding convolution kernels assigned to the slave processing circuits, respectively.

[0248] Clause 2. The computing device according to Clause 1, wherein the input channel splitting granularity Pci is determined based on the size of the input channel Ci dimension of the input feature map and the M.

[0249] Clause 3. The computing device according to Clause 2, wherein the Pci is determined as follows:

[0250] Align the size Ci of the input channel dimension of the input feature map to a plurality of predetermined alternative splitting granularities respectively; and

[0251] Select, as the Pci, an alternative splitting granularity with an alignment padding amount within a predetermined range and as large as possible according to the alignment padding amount required for alignment to each alternative splitting granularity and the size of the corresponding splitting granularity, where the alternative splitting granularities are selected from M / 2 n , n = 0, 1, 2, ….

[0252] Clause 4. The computing device according to any one of Clauses 1-3, wherein the storage circuit includes a first storage circuit, and the input feature map is stored in the first storage circuit in the original format.

[0253] Clause 5. The computing device according to Clause 4, wherein the storage circuit further includes a second storage circuit, and the convolution kernel is stored in blocks according to the value of the output channel Co in the storage areas allocated to each slave processing circuit in the second storage circuit, where each slave processing circuit is allocated a convolution kernel for processing different Aco / Ns Co values at an interval of Ns, where Ns is the number of schedulable slave processing circuits and Aco is the output channel alignment value.

[0254] Clause 6. The computing device according to Clause 5, wherein the output channel alignment value Aco is determined as follows:

[0255] When Ns*(m - 1) < Co ≤ Ns*m, m = 1, 2, …, Aco = Ns*m.

[0256] Clause 7. The computing device according to any one of Clauses 5-6, wherein each slave processing circuit includes a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, where:

[0257] The first buffer circuit is used to cache a plurality of input feature data rows transmitted by broadcast from the first storage circuit;

[0258] The second buffer circuit is used to cache a plurality of extended weight data rows of the convolution kernel distributed to the slave processing circuit from the second storage circuit; and

[0259] Each arithmetic circuit is used to perform a bitwise multiply-accumulate operation on the input feature data row selected from the first buffer circuit and the extended weight data row selected from the second buffer circuit in each operation.

[0260] Clause 8. The computing device according to Clause 7, wherein the main processing circuitry is further configured to:

[0261] When Ci does not exceed Pci, continuous input feature map data of Ci*Ws is read from the first storage circuit, and after selectively performing alignment and padding processing, it is broadcast as a data row to multiple slave processing circuits so as to continuously write the data row into the first buffer circuit of the slave processing circuit.

[0262] Clause 9. The computing device according to Clause 7, wherein the main processing circuitry is further configured to:

[0263] When Ci exceeds Pci, Ws intervals of input feature map data blocks are read from the first storage circuit. After selectively performing alignment and padding processing, they are broadcast as a data line to multiple slave processing circuits to continuously write the data line into the first buffer circuit of the slave processing circuit. Each input feature map data block is split or aligned to Pci in the Ci dimension, and the starting address of each read input feature map data block is sequentially spaced Ci.

[0264] Clause 10. The computing device according to Clause 7 further includes a first intermediate storage circuit, and the main processing circuit is further configured to:

[0265] When Ci exceeds Pci, continuous input feature map data of Ci*Ws is read from the first storage circuit and continuously written into the first intermediate storage circuit; and

[0266] Ws intervals of input feature map data blocks are read from the first intermediate storage circuit. After selective alignment and padding processing, they are broadcast as a data line to multiple slave processing circuits to continuously write the data line into the first buffer circuit of the slave processing circuit. Each input feature map data block is split or aligned to Pci in the Ci dimension, and the starting address of each read input feature map data block is sequentially spaced Ci.

[0267] Clause 11. A computing device according to any one of Clauses 7-10, wherein:

[0268] When the convolution kernel in the second storage circuit is distributed to the second buffer circuit of each slave processing circuit, the input channel dimension of the convolution kernel is aligned to the Pci, and according to the multiple of the width dimension Ws, a column of input channel data blocks split or aligned to Pci in the Ci dimension is copied and expanded into Ws columns to form an expanded weight data row, which is then transmitted to the second buffer circuit of the corresponding slave processing circuit.

[0269] Clause 12. The computing device according to Clause 11, wherein each of the said slave processing circuits is further configured to perform the convolution operation in a dimensional order, in multiple layers of loops, wherein:

[0270] Each Co value allocated from the processing circuit is used as the innermost loop, and the number of loops Nco = Aco / Ns;

[0271] The Kw dimension of the convolution kernel is used as the middle layer loop, and the number of loops is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel and Kmax is the maximum convolution kernel width value supported by the processing circuit.

[0272] The convolution kernel's Kh and Ci dimensions are divided into blocks Bci according to Pci, which is used as the outer loop. The number of loops is Nkh = Kh, and Nci = Bci = ceil(Ci / Pci), where Kh is the height dimension of the convolution kernel; and

[0273] The total number of cycles, Ncycle, is equal to Nco * Nkw * Nkh * Nci.

[0274] Clause 13. The computing device according to Clause 12, wherein in the outer loop,

[0275] First loop along the Kh dimension, then loop along the block count along the Ci dimension; or

[0276] First, iterate through the number of blocks in the Ci dimension, then iterate through the Kh dimension.

[0277] Clause 14. The computing device according to any one of Clauses 12-13, wherein for each Co value of the innermost loop, each of the said processing circuits is further configured to:

[0278] It utilizes its internally schedulable Ncu arithmetic circuits to compute in parallel Ncu output blocks that are consecutive in the width Wo dimension of the output feature map, with each output block comprising Ws output points that are consecutive in the width Wo dimension.

[0279] Clause 15. The computing device according to Clause 14, wherein each of the said slave processing circuits is further configured to:

[0280] Ncu adjacent rows of input feature data are selected from the first buffer circuit and distributed to the Ncu arithmetic circuits for calculation.

[0281] Select a corresponding extended weight data row from the second buffer circuit and broadcast it to the Ncu arithmetic circuits;

[0282] At the Ncu arithmetic circuits, for the distributed input feature data lines and the broadcast extended weight data lines, bitwise multiplication and accumulation are performed in units of 1 / Ws data lines to obtain a partial sum of Ws output points.

[0283] Clause 16. The computing device according to any one of Clauses 12-15, wherein in the innermost loop, each input feature data line is multiplexed Aco / Ns times to compute Ws output points on the same width Wo dimension of Aco / Ns Co values ​​on the output feature map.

[0284] Clause 17. The computing device according to any one of Clauses 12-15, wherein in the middle loop of the Kw dimension, the input feature data row and the extended weight data row are synchronously selected in the width dimension by sliding with a step size of 1 on the first buffer circuit and the second buffer circuit.

[0285] Clause 18. In any of the computing devices described in Clauses 12-17, the maximum convolutional kernel width value Kmax supported by the processing circuitry is determined as follows:

[0286] Kmax = L1*Ws - Ncu*Ws+1, where L1 is the size of the first buffer circuit in data rows; Ncu is the number of scheduled arithmetic circuits.

[0287] Clause 19. The computing device according to any one of Clauses 12-18 further includes a third buffer circuit, wherein each of the slave processing circuits is further configured to:

[0288] The results of the calculations from each arithmetic circuit are stored in the third buffer circuit in the order of width Wo first, then Co dimension; and

[0289] The computation results are output from the third buffer circuit in the order of Co dimension first, followed by Wo dimension.

[0290] Clause 20. The computing device according to Clause 19, wherein the main processing circuitry is further configured to:

[0291] The calculation results output from each of the processing circuits are concatenated and stored according to the HoWoCo dimensional storage order.

[0292] Clause 21. The computing device according to Clause 20, wherein the main processing circuitry is further configured to:

[0293] When storing the computation results output from the processing circuit, output points in the Wo dimension that are redundantly computed due to alignment calculations are filtered out.

[0294] Clause 22. The computing device according to any one of Clauses 20-21, wherein the main processing circuitry is further configured to:

[0295] When storing the computation results output from the processing circuit, output points in the Co dimension that are redundantly calculated due to alignment are filtered out.

[0296] Clause 23. The computing device according to Clause 22, wherein the main processing circuitry is further configured to:

[0297] The output feature map is split into output channel Co dimension according to the output channel splitting granularity Pco, and the data of each splitting granularity is combined with the data of the width dimension Wso times to form the output feature data row for transmission, where Pco*Wso=M.

[0298] Clause 24. The computing device according to Clause 23, wherein the main processing circuitry is further configured to:

[0299] When Co does not exceed Pco, read the output feature map data of size Co*Wso from the output calculation result; and

[0300] The read output feature map data is continuously written into the first storage circuit.

[0301] Clause 25. The computing device according to Clause 23, wherein the main processing circuitry is further configured to:

[0302] When Co exceeds Pco, the Co dimension is divided into an integer part and a remainder part according to Pco. For the integer part, output feature map data of size Pco*Wso is read from the output calculation result each time. This output feature map data is divided into Wso output feature map data blocks, each of which includes data of size Pco in the Co dimension; and

[0303] The Wso output feature map data blocks are written into the first storage circuit at intervals, with the starting address of each written output feature map data block being sequentially spaced Co apart.

[0304] Clause 26. The computing device according to Clause 25, wherein the main processing circuitry is further configured to:

[0305] When Co exceeds Pco, for the remainder portion, output feature map data of size Co_rem*Wso is read from the output calculation result. This output feature map data is divided into Wso output feature map data blocks, each block including data of size Co_rem in the Co dimension, where Co_rem represents the remainder obtained by dividing Co by Pco; and

[0306] The Wso output feature map data blocks are written into the first storage circuit at intervals, with the starting address of each written output feature map data block being sequentially spaced Co apart.

[0307] Clause 27. A chip comprising a computing device according to any one of Clauses 1-26.

[0308] Clause 28. A board card including the chip described in Clause 27.

[0309] Clause 29. A method for performing convolution operations using any of the computing devices described in Clauses 1-26.

[0310] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A computing device configured to perform a convolution operation, the computing device comprising: A storage circuit that stores input feature maps and convolutional kernels, wherein the convolutional kernels are stored in blocks according to the output channel Co dimension for allocation to different slave processing circuits; The main processing circuit aligns the input channel Ci dimension of the input feature map to the input channel splitting granularity Pci, and combines the data of each splitting granularity with the data of a width dimension Ws times to form an input feature data row, which is then broadcast to multiple slave processing circuits to perform the convolution operation, wherein Pci Ws = M, where M is the amount of data processed in a single hardware operation; and multiple slave processing circuits, each configured to perform convolution operations on the broadcast input feature map and the corresponding convolution kernels assigned to the slave processing circuits. The input channel splitting granularity Pci is determined based on the size of the input channel Ci dimension of the input feature map and M.

2. The computing device according to claim 1, wherein the Pci is determined as follows: aligning the size Ci of the input channel dimension of the input feature map to a predetermined plurality of candidate split granularities; and selecting, based on the alignment padding amount required to align to each candidate split granularity and the size of the corresponding split granularity, a candidate split granularity with an alignment padding amount within a predetermined range and as large as possible as the Pci, wherein the candidate split granularity is selected from M / 2. n , n = 0, 1, 2, ...

3. The computing device according to any one of claims 1-2, wherein the storage circuit includes a first storage circuit, and the input feature map is stored in the first storage circuit in its original format.

4. The computing device according to claim 3, wherein the storage circuit further comprises a second storage circuit, wherein the convolution kernels are stored in blocks according to the output channel Co value in the storage area allocated to each slave processing circuit in the second storage circuit, wherein each slave processing circuit is assigned to process convolution kernels with different Aco / Ns Co values ​​spaced at Ns intervals, wherein Ns is the number of schedulable slave processing circuits and Aco is the output channel alignment value.

5. The computing device according to claim 4, wherein the output channel alignment value Aco is determined as follows: when Ns (m-1) <Co≤Ns When m = 1, 2, ..., Aco = Ns m.

6. The computing device according to any one of claims 4-5, wherein each of the slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein: The first buffer circuit is used to buffer multiple lines of input feature data that are broadcast from the first storage circuit; The second buffer circuit is used to cache multiple extended weight data rows from the second storage circuit that are distributed to the convolution kernel of the processing circuit; and each operation circuit is used to perform bitwise multiplication and accumulation operations in each operation for the input feature data row selected from the first buffer circuit and the extended weight data row selected from the second buffer circuit, respectively.

7. The computing device of claim 6, wherein the main processing circuit is further configured to: read consecutive Ci values ​​from the first storage circuit when Ci does not exceed Pci. The input feature map data of Ws is selectively aligned and padded, and then broadcast as a data row to multiple slave processing circuits to continuously write the data row into the first buffer circuit of the slave processing circuit.

8. The computing device of claim 6, wherein the main processing circuit is further configured to: when Ci exceeds Pci, read Ws intervals of input feature map data blocks from the first storage circuit, selectively perform alignment and padding processing, and broadcast them as a data line to a plurality of the slave processing circuits to continuously write the data line into the first buffer circuit of the slave processing circuit, wherein each input feature map data block is split or aligned to Pci in the Ci dimension, and the starting address of each read input feature map data block is sequentially spaced Ci.

9. The computing device of claim 6, further comprising a first intermediate storage circuit, and the main processing circuit further configured to: when Ci exceeds Pci, read consecutive Ci values ​​from the first storage circuit. The input feature map data Ws is continuously written into the first intermediate storage circuit; and Ws intervals of input feature map data blocks are read from the first intermediate storage circuit, and after selectively performing alignment and padding processing, they are broadcast as a data line to multiple slave processing circuits to continuously write the data line into the first buffer circuit of the slave processing circuit, wherein each input feature map data block is split or aligned to Pci in the Ci dimension, and the starting address of each input feature map data block read is sequentially spaced Ci.

10. The computing device according to claim 6, wherein: When the convolution kernel in the second storage circuit is distributed to the second buffer circuit of each slave processing circuit, the input channel dimension of the convolution kernel is aligned to the Pci, and according to the multiple of the width dimension Ws, a column of input channel data blocks split or aligned to Pci in the Ci dimension is copied and expanded into Ws columns to form an expanded weight data row, which is then transmitted to the second buffer circuit of the corresponding slave processing circuit.

11. The computing device of claim 10, wherein each of the slave processing circuits is further configured to perform the convolution operation sequentially and in multiple layers, wherein: Each Co value allocated by the processing circuit is used as the innermost loop, with a loop count Nco = Aco / Ns; the Kw dimension of the convolutional kernel is used as the middle loop, with a loop count Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolutional kernel, and Kmax is the maximum convolutional kernel width supported by the processing circuit; the number of blocks Bci of the Kh and Ci dimensions of the convolutional kernel split according to Pci is used as the outer loop, with a loop count Nkh = Kh, Nci = Bci = ceil(Ci / Pci), where Kh is the height dimension of the convolutional kernel; and the total number of loops Ncycle = Nco Nkw Nkh Nci.

12. The computing device according to claim 11, wherein in the outer loop, the Kh-dimensional loop is performed first, followed by the Ci-dimensional block loop; or the Ci-dimensional block loop is performed first, followed by the Kh-dimensional loop.

13. The computing device according to any one of claims 11-12, wherein for each Co value of the innermost loop, each of the slave processing circuits is further configured to: compute in parallel Ncu output blocks that are consecutive in the width Wo dimension of the output feature map using Ncu schedulable arithmetic circuits therein, each output block comprising Ws output points that are consecutive in the width Wo dimension.

14. The computing device of claim 13, wherein each of the slave processing circuits is further configured to: select Ncu adjacent input feature data rows from the first buffer circuit and distribute them to the Ncu arithmetic circuits for computation; select a corresponding extended weight data row from the second buffer circuit and broadcast it to the Ncu arithmetic circuits; and at the Ncu arithmetic circuits, perform bitwise multiplication and accumulation on the distributed input feature data rows and the broadcast extended weight data rows in units of 1 / Ws data rows to obtain a partial sum of Ws output points.

15. The computing apparatus of claim 14, wherein in the innermost loop, each input feature data line is multiplexed Aco / Ns times to compute Ws output points on the same width Wo dimension of Aco / Ns Co values ​​on the output feature map.

16. The computing device of claim 14, wherein in the intermediate loop of the Kw dimension, the input feature data row and the extended weight data row are synchronously selected in the width dimension by sliding with a step size of 1 on the first buffer circuit and the second buffer circuit.

17. The computing device of claim 16, wherein the maximum convolutional kernel width Kmax supported by the slave processing circuit is determined as follows: Kmax = L1 Ws-Ncu Ws+1, where L1 is the size of the first buffer circuit in data rows; Ncu is the number of scheduled arithmetic circuits.

18. The computing device of claim 17, further comprising a third buffer circuit, wherein each of the slave processing circuits is further configured to: store the operation results of each arithmetic circuit in the third buffer circuit in the order of width Wo dimension first, then Co dimension; and output the operation results from the third buffer circuit in the order of Co dimension first, then Wo dimension.

19. The computing device of claim 18, wherein the main processing circuit is further configured to: concatenate and store the computation results output from each of the secondary processing circuits in the order of HoWoCo dimensional storage.

20. The computing device of claim 19, wherein the main processing circuit is further configured to: filter out output points in the Wo dimension that are redundantly calculated due to alignment calculations when storing the computation result output from the slave processing circuit.

21. The computing device according to any one of claims 19-20, wherein the main processing circuit is further configured to: filter out output points in the Co dimension that are redundantly calculated due to alignment when storing the computation result output from the slave processing circuit.

22. The computing device of claim 21, wherein the main processing circuit is further configured to: split the output channel Co dimension of the output feature map according to the output channel splitting granularity Pco, and combine the data of each splitting granularity with the data of the width dimension Wso times to form an output feature data row for transmission, wherein Pco Wso = M.

23. The computing device according to claim 22, wherein the main processing circuit is further configured to: read Co from the output calculation result when Co does not exceed Pco. Output feature map data of size Wso; and continuously write the read output feature map data into the first storage circuit.

24. The computing device of claim 22, wherein the main processing circuit is further configured to: when Co exceeds Pco, divide the Co dimension into an integer part and a remainder part according to Pco, and for the integer part, read Pco from the output calculation result each time. The output feature map data is of size Wso, wherein the output feature map data is divided into Wso output feature map data blocks, each output feature map data block includes data of size Pco in the Co dimension; and the Wso output feature map data blocks are written into the first storage circuit at intervals, with the starting address of each written output feature map data block sequentially spaced Co apart.

25. The computing device of claim 24, wherein the main processing circuit is further configured to: when Co exceeds Pco, for the remainder portion, read Co_rem from the output calculation result. The output feature map data is of size Wso, wherein the output feature map data is divided into Wso output feature map data blocks, each output feature map data block includes data of size Co_rem in the Co dimension, where Co_rem represents the remainder obtained by dividing Co by Pco; and the Wso output feature map data blocks are written into the first storage circuit at intervals, with the starting address of each written output feature map data block being sequentially spaced Co apart.

26. A chip comprising a computing device according to any one of claims 1-25.

27. A circuit board comprising the chip according to claim 26.

28. A method for performing convolution operations using a computing device according to any one of claims 1-25.

Images