Computing devices, methods, and related products for performing convolution operations

Abstract

This disclosure discloses a computing device, a method for performing convolution operations using the computing device, and related products. The computing device may be included in a combined processing unit, which may further include an interface device and other processing units. The computing device interacts with the other processing units to jointly complete user-specified computational operations. The combined processing unit may also include a storage device connected to both the computing device and the other processing units to store data from them. The disclosed solution optimizes convolution operations, improves data reuse efficiency, and thereby enhances computational processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a computing device, a method for performing convolution operations using the computing device, a chip, and a circuit board. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Neural networks are among the most critical technologies in artificial intelligence and deep learning, with Convolutional Neural Networks (CNNs) being the most important type. The most crucial computation in CNNs is the convolution operation within the convolutional layers (Conv layers). The function of convolutional layers is to extract features from input data. Through multiple convolutions, complex features can be extracted to ensure the network has sufficient expressive and generalization capabilities. Neural network models contain numerous convolution operations of various types, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. When neural network models are applied to different fields, such as speech recognition, machine translation, and image processing, the corresponding input feature maps and the dimensions of the weights may vary. To fully utilize the hardware advantages of deep learning processors, optimization is needed for convolution operations of different scales and / or types to improve the computational performance of the neural network model. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes a computing device in several aspects that, by folding the width dimension of the input feature map, enables data of various dimensional sizes to be adapted to the hardware of convolution operations, thereby improving the computational efficiency of convolution operations. The convolution operations in embodiments of this disclosure can be operations in various neural network models applicable to various fields, such as image processing, speech processing, text processing, etc., and these processes may include, but are not limited to, recognition and classification.

[0005] In a first aspect, embodiments of this disclosure provide a computing device including multiple slave processing circuits, each slave processing circuit including a first buffer circuit, a second buffer circuit, and multiple arithmetic circuits, wherein: the first buffer circuit is used to buffer multiple input feature rows for which convolution operations will be performed, wherein one input feature row includes a data volume of Pci×Ws=M in the input feature map, where Pci is the splitting granularity of the input channel Ci dimension, Ws is the folding factor of the width W dimension, and M is the amount of data processed by the hardware in a single operation; the second buffer circuit is used to buffer weight data for which convolution operations will be performed; and each of the arithmetic circuits is used to perform a bitwise multiplication and accumulation operation on the input feature rows selected from the first buffer circuit and the extended weight rows selected or generated from the second buffer circuit respectively during each calculation, wherein one extended weight row is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolution kernel into a column of Ws.

[0006] In a second aspect, embodiments of this disclosure provide a chip that includes the computing device described in the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip described in the second aspect.

[0008] Based on the computing device, chip, board, and method for performing convolution operations by the computing device provided above, the scheme of this disclosure applies different width-dimension folding schemes to input feature maps of different dimensions to adapt to the processing capabilities of the hardware computing device, thereby fully utilizing the parallel processing capabilities of multiple processing circuits and effectively improving the computational efficiency of convolution operations. Furthermore, weights can be reused at a granularity of less than one weight row, thereby reducing frequent data loading and improving computational efficiency. Other advantages and effects will become readily apparent from the detailed description following in conjunction with the accompanying drawings. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0010] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0011] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0012] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0013] Figure 4 This illustrates an exemplary example of the convolution operation principle that can be applied to the embodiments disclosed herein;

[0014] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0015] Figures 6a-6c Several examples of data width dimension folding according to embodiments of this disclosure are shown;

[0016] Figure 7 The illustration shows a schematic storage method of an input feature map according to some embodiments of this disclosure;

[0017] Figure 8 A schematic diagram of the convolution kernel storage method according to an embodiment of this disclosure is shown;

[0018] Figure 9 An exemplary loop diagram illustrating the computation of a single convolution output point according to an embodiment of this disclosure is shown;

[0019] Figure 10 This diagram illustrates a computational schematic of reusing input feature map data along the H dimension according to some embodiments of this disclosure;

[0020] Figure 11 A schematic splitting method of the output feature map according to an embodiment of this disclosure is shown;

[0021] Figures 12a-12c A schematic diagram illustrating the operation process of the convolution operation scheme according to an embodiment of this disclosure is shown; and

[0022] Figure 13 A schematic diagram illustrating the logic for writing and outputting computation results according to an embodiment of this disclosure is shown. Detailed Implementation

[0023] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0025] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0026] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0027] Exemplary hardware environment

[0028] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0029] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0030] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0031] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0032] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0033] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0034] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0035] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0036] Figure 3 The diagram shows the internal structure of the processing core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0037] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0038] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0039] The storage module 33 is used to store or move relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. The NRAM 331 stores the input neurons, output neurons, and intermediate results after computation; the WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; the DMA 333 is connected to the DRAM 204 via bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204.

[0040] Exemplary convolution operation types

[0041] Based on the aforementioned hardware environment, in one aspect, this disclosure provides a computing device configured to perform convolution operations, thereby optimizing convolution operations, for example, in neural network models. Convolutional layers in a neural network model can perform convolution operations by applying convolution kernels (also called filters, weights, etc.) to an input feature map (also called input data, neurons, or input neurons) to extract features. A convolutional layer may contain multiple convolution kernels, each element of which corresponds to a weight coefficient and a bias.

[0042] Neural network models may contain various convolutional layers, such as convolutional layers that perform forward, regular 3D convolutional operations, and deconvolutional layers that perform depthwise convolutional operations. In reverse training, it may be necessary to perform inverse depthwise convolutional operations or cross-product convolutional operations. The embodiments disclosed herein are primarily optimized for regular 3D convolutional operations, but can also be applied to other types of convolutional operations without conflict.

[0043] In conventional 3D convolution operations, assuming the input feature map tensor shape in the convolutional layer is represented by X[N Hi Wi Ci], the tensor shape of the convolutional kernel is represented by K[Co Kh Kw Ci], and the output is Y[N HoWo Co], then the simplified mathematical formula for convolution operations can be expressed as follows:

[0044] Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×Kjc,ic,ih,iw (1)

[0045] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, and sh and sw are the strides in the length and width directions. The formula ignores the bias, padding, and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N and C dimensions. The forward computation of the neural network model is independent in the N dimension, but fully connected in the C dimension. When the convolution kernel works, it scans the input features with a certain stride, performs matrix element-wise multiplication and sums the bias within the convolution window. In conventional 3D convolution operations, the results of the alignment multiplications in the H, W, and Ci directions are accumulated, hence the name 3D convolution. However, this 3D convolution has a constraint: the Ci dimension of the convolution kernel is equal to the Ci dimension of the input feature map, so the convolution kernel does not slide in the Ci direction, making it a pseudo-3D convolution. For simplicity, the above convolution operation is called 3D convolution operation.

[0046] Figure 4 An example of a conventional 3D convolution operation principle that can be applied to embodiments of this disclosure is shown.

[0047] The figure exemplifies a four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N Hi×Wi×Ci 3D rectangles 410. The figure also exemplifies a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co Kh×Kw×Ci 3D convolution kernels 420. The convolution result of the input data X and the convolution kernel K yields the output data Y, which is a four-dimensional data of size [N Ho Wo Co], which can be represented as N Ho×Wo×Co 3D rectangles 430.

[0048] The figure also shows a specific example of a convolution operation, where the input data is a 6×6×3 input feature map 440, omitting the N dimension; the convolution kernel is a 3×3×3 stereo convolution kernel 450, targeting a single Co; and the output data is a 4×4 output feature map 460. The specific operation process is as follows:

[0049] The convolution kernel 450 scans across the input feature map 440a with a certain stride, performs element-wise matrix multiplication on the input features within the convolution window 470, and then sums the bias values. That is, the value at each position in the output feature map 460 is obtained by performing a two-dimensional convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) in the output feature map 460 (i.e., the convolution output point) is obtained by performing a two-dimensional convolution operation between the convolution window 470 (framed by the black cube in the input feature map) and the 3D convolution kernel 450, resulting in three values, which are then summed to obtain the final value.

[0050] To obtain outputs at other locations, the position of the convolution kernel 450 can be moved on the input feature map 440, which means moving the convolution window of the output point. In the example in the figure, the convolution stride (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0,1) or (1,0) on the output feature map 460a can be obtained respectively.

[0051] As described above, a convolutional layer in a neural network contains N sets of input feature maps, each containing Hi × Wi × Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output feature map contains Ho × Wo × Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation also involves the convolution stride (Sx, Sy), the size of which affects the size of the output feature map.

[0052] In this paper, the input feature map, input data, neuron, or input neuron can be used interchangeably; the convolution kernel, filter, or weights can be used interchangeably; and the output feature map, output data, or output neuron can be used interchangeably. Furthermore, the H (height) and Y dimensions can be used interchangeably, as can the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In the embodiments disclosed herein, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window.

[0053] Exemplary computing device

[0054] In this disclosed embodiment, a master-slave computing device can be used to perform the above-described convolution operation. Furthermore, different data paths can be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0055] Figure 5 A schematic structural block diagram of a computing device 500 according to an embodiment of this disclosure is shown. It can be understood that this structure can be considered as... Figure 3 The detailed internal structure of the processing module of a single processing core can also be viewed as a refinement of multiple processing cores. Figure 3 The diagram shows a functional partitioning based on the processing core's computational modules. Figure 5 As shown, the computing device 500 of this disclosure embodiment can be configured to perform various types of convolution operations, and may include a main processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520. Sixteen slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will understand that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and this disclosure embodiment is not limited in this respect.

[0056] The master processing circuit and slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections. In different application scenarios, the connection between multiple slave processing circuits can be either a hard connection arranged by hardwired lines or a logical connection configured according to, for example, microinstructions, to form a topology of various slave processing circuit arrays. The embodiments disclosed herein are not limited in this respect. The master processing circuit and slave processing circuits can cooperate with each other to achieve parallel processing.

[0057] To support computational functions, the main processing circuit and the slave processing circuit can include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit is responsible for the core computations of deep learning algorithms, such as matrix multiplication and convolution.

[0058] The processing circuit can, for example, perform intermediate operations on the corresponding data in parallel according to the operation instructions to obtain multiple intermediate results, and then transmit the multiple intermediate results back to the main processing circuit.

[0059] By configuring the computing device 500 into a master-slave structure (e.g., a master-multiple-slave structure, or a multi-master-multiple-slave structure, which is not limited in this disclosure), for forward computation instructions, the data can be split according to the computation instructions, thereby enabling multiple slave processing circuits to perform parallel computation on the computationally intensive parts to improve computation speed, save computation time, and thus reduce power consumption.

[0060] In some embodiments disclosed herein, by utilizing different data paths to transmit input feature maps and weights, multiple reuse methods for input feature maps and weights can be supported, thereby reducing data access during computation and improving processing efficiency.

[0061] Specifically, the computing device 500 may further include a first storage circuit 530 and a second storage circuit 540 for storing data transmitted via different data channels, respectively. Optionally, the first storage circuit 530 and the second storage circuit 540 may be two storage blocks formed by partitioning the same memory, or they may be two independent memories; no specific limitation is made here.

[0062] The first storage circuit 530 can be used to store multicast data, meaning that the data in the first storage circuit will be transmitted to multiple slave processing circuits via a broadcast bus, and these slave processing circuits will receive the same data. It can be understood that broadcasting and multicasting can be implemented via a broadcast bus. Multicast refers to a communication method that transmits a single data set to multiple slave processing circuits; while broadcasting is a communication method that transmits a single data set to all slave processing circuits, and is a special case of multicast. Since both multicast and broadcasting correspond to one-to-many transmission methods, this document does not specifically distinguish between the two; broadcasting and multicast can be collectively referred to as multicast, and those skilled in the art can understand their meaning from the context.

[0063] The second storage circuit 540 can be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0064] By providing a first storage circuit and a second storage circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by multiplexing multicast data among multiple slave processing circuits.

[0065] In some embodiments, the input feature map can be determined as multicast data and stored in a first storage circuit to transmit the data via broadcast to multiple scheduled slave processing circuits during computation. Correspondingly, the convolution kernel can be determined as distribution data and stored in a second storage circuit. This distribution data can be distributed to the corresponding slave processing circuits before computation.

[0066] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of this disclosure is also shown. As shown, each slave processing circuit 520 may include multiple arithmetic circuits CU 521, a first buffer circuit 522, and a second buffer circuit 523. Four arithmetic circuits CU0 to CU3 are shown in the figure. Those skilled in the art will understand that the number of arithmetic circuits may be more or less, depending on the specific hardware configuration, and the embodiments of this disclosure are not limited in this respect.

[0067] In some embodiments, the first buffer circuit 522 can be used to buffer the weights or input feature maps allocated to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to buffer the input feature maps or weights allocated to the slave processing circuit. Both buffer circuits are used to select data for computation. The data in the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540; correspondingly, the data in the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computation circuit CU 521 or broadcast to all CUs 521 within the slave processing circuit 520 during computation.

[0068] Each arithmetic circuit CU 521 is used to perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit and data rows selected from the second buffer circuit, respectively, in each arithmetic cycle.

[0069] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by reusing data as much as possible among multiple processing circuits within a single slave processing circuit.

[0070] The processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each arithmetic circuit CU 521.

[0071] Understandable, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but depending on the configuration, the storage circuits and processing circuits can also be combined into a single module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, while the second storage circuit 540 can be shared by multiple slave processing circuits 520, with each slave processing circuit allocated an independent storage area to accelerate access. This disclosure does not limit the embodiments in this respect. Furthermore, in this computing device, the main processing circuit and slave processing circuits can belong to different modules of the same processor or chip, or they can belong to different processors; this disclosure also does not limit this in this respect.

[0072] Exemplary Convolution Optimization Scheme

[0073] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is typically allocated in contiguous storage space, meaning it can be unfolded one-dimensionally and stored sequentially in memory. For example, in this disclosed embodiment, the input feature map can be stored sequentially according to a low-dimensional priority (here, C / Ci is the lowest dimension). Adjacent dimensions refer to dimensions that are immediately next to each other in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. When the storage order is consistent with the dimension order, adjacent dimensions are located consecutively in memory. Here, W and Ci are adjacent, and their data is also consecutively stored in memory.

[0074] In intelligent processors, due to computational power requirements and considerations of area and power consumption, the main hardware operation unit is the vector multiply-accumulate unit. Implementing support for various convolution algorithms in hardware design essentially involves maximizing the extraction of multiply-accumulate operations from the algorithms and implementing them in on-chip RAM (such as...) via data paths. Figure 3 It efficiently exchanges input and output data for multiplication and addition operations between NRAM, WRAM, etc. and the arithmetic unit.

[0075] Hardware stores data line by line (cache line). Read, write, and computation operations are most efficient when aligned to the entire line. Therefore, to fully utilize bandwidth and accommodate the memory access demands of the arithmetic unit array, data is typically vectorized and aligned. Artificial intelligence chips are usually designed with the Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above, where data along the Ci dimension is continuous. Therefore, vectorization alignment requires the Ci dimension to be aligned to a specified value, such as alignment value M, so that data can be accessed in units of this alignment value M. M can also be called the maximum number of operations per hardware operation. Depending on the hardware design, M can have different values, such as 64 bits, 128 bits, 256 bits, 512 bits, etc. Typically, the size of the arithmetic unit array's input ports is also related to M. For example, in the case of symmetrical input data bit width, the input port size of the arithmetic unit array is usually twice M, meaning it can process input feature map data and weight data of alignment value M in one operation. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0076] When the Ci dimension of the input feature map is small, or when the remainder obtained by dividing Ci by M is small (e.g., less than the size of a cache line), the Ci dimension needs to be padded to a single line of data (e.g., 512 bits), i.e., filled with invalid data 0. This padding causes a large amount of redundant computation, resulting in wasted resources and reduced computational efficiency.

[0077] A small convolution scheme suitable for cases with a small channel C has been proposed, in which the computational data is split into split units and stored in a transformed dimensional order. The amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, thereby performing computational processing on a unit-by-unit basis, which can fully utilize the hardware's computing power and avoid or reduce invalid computations.

[0078] However, in this small convolution scheme, both the input feature map and the convolution kernel need to be pre-processed by software through block division and dimensionality transformation, and the output feature map also needs to be correspondingly processed by software through block division and dimensionality transformation, which undoubtedly increases the complexity of the software. Furthermore, alignment processing is also required during these block division and dimensionality transformation processes. Moreover, these small convolution schemes only support convolution operations with a stride of 1 in both the width and height directions.

[0079] In view of this, in order to further optimize convolution operations and reduce software complexity, this disclosure provides a convolution scheme with width-dimensional folding, which compensates the data of the width W dimension, which is continuous with the input channel Ci dimension of the input feature map, to the Ci dimension only when needed, thus eliminating the need for software to perform data block and dimension transformation processing.

[0080] Specifically, in some embodiments, a computing device is provided, including multiple slave processing circuits. Each slave processing circuit includes a first buffer circuit, a second buffer circuit, and multiple arithmetic circuits, wherein: the first buffer circuit is used to buffer multiple input feature rows for which convolution operations will be performed, wherein one input feature row includes a data volume of Pci×Ws=M in the input feature map, where Pci is the splitting granularity of the input channel Ci dimension, Ws is the folding factor of the width W dimension, and M is the amount of data processed by the hardware in a single operation; the second buffer circuit is used to buffer the weight data for which convolution operations will be performed; and each arithmetic circuit is used to perform a bitwise multiplication and accumulation operation on the input feature rows selected from the first buffer circuit and the extended weight rows selected or generated from the second buffer circuit respectively during each calculation, wherein one extended weight row is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolution kernel into a column Ws.

[0081] In some embodiments, the output data of the layer above certain convolutional layers (e.g., FUCONV) has been divided into two segments in the Ci dimension, with each segment having a ci size of 32B (e.g., data type int8) or 64B (e.g., data type int16). In this case, the splitting granularity Pci can follow the size of each segment, i.e., 32B or 64B.

[0082] In some embodiments, the input channel splitting granularity Pci can be determined based on the size of the input channel dimension Ci of the input feature map and the amount of data M processed by the hardware in a single operation. Then, the folding factor Ws of the width W dimension of the input feature map can be determined based on the splitting granularity Pci. In some embodiments, Ws = M / Pci. It is understood that the convolution scheme of this disclosure embodiment, by splitting the Ci dimension according to the splitting granularity, can be suitable for any Ci size. Furthermore, it is understood that the maximum splitting granularity Pci does not exceed the hardware's single-operation alignment value M (or baseline alignment value, the amount of data processed by the hardware in a single operation). Therefore, under different value ranges of Ci, a suitable Pci can be selected, and by filling the data in adjacent W dimensions into the Ci dimension, the alignment requirements for the Ci dimension can be reduced.

[0083] In some embodiments, the input channel splitting granularity Pci can be selected as M / 2. n n = 0, 1, 2, ..., thus facilitating storage from the second lowest dimension W in 2... n Fold the data to the lowest storage dimension Ci. Table 1 shows several exemplary folding schemes corresponding to the input channel splitting granularity Pci, assuming M = 64B.

[0084] Split granularity (Pci) 4B 8B 16B 32B 64B Ws (W fold) 16 8 4 2 1

[0085] Table 1

[0086] As can be seen from Table 1, the smaller the input channel splitting granularity, the more parts Wi supplies in the Ci direction, and the larger the alignment constraint on Wi, which needs to satisfy Wi / Ws≥1.

[0087] This is understandable, although theoretically the granularity of the split can be M / 2. n However, considering factors such as the requirements for the W dimension, instruction overhead, and the actual range of Ci values ​​when the splitting granularity is too small, it is advisable to choose only M / 2. n A subset of the values ​​in the table are used as alternative splitting granularities. In the example where M = 64B, alternative splitting granularities could include, for example, 64B, 32B, 16B, and 8B.

[0088] Different splitting granularities can be applied to different computational scenarios, thereby achieving varying degrees of performance optimization. Specifically, in some embodiments, the input channel splitting granularity Pci can be selected as follows:

[0089] Align the lowest storage dimension Ci of the input feature map to each of the candidate splitting granularities; and

[0090] Taking into account the alignment padding amount and the size of the corresponding splitting granularity for each candidate splitting granularity, a suitable splitting granularity is selected, such as a candidate splitting granularity with an alignment padding amount within a predetermined range and as large as possible, as the Pci.

[0091] For example, when the alignment fill amount is the same, a larger split granularity is preferred; or when the alignment fill amount is different, a split granularity with the smallest alignment fill amount is selected; or when the alignment fill amounts are not significantly different (e.g., within a predetermined range, such as no more than 16B), a larger split granularity is preferred.

[0092] Although the rules for selecting the input channel splitting granularity Pci are listed above, these rules are merely preferred embodiments for selecting the optimal input channel splitting granularity best suited to the current Ci value. The application of these rules is described below with several examples. All examples assume M = 64B, and alternative splitting granularities include 64B, 32B, 16B, and 8B.

[0093] In one example, assuming Ci = 48B, alignment to 8B and 16B requires no zero padding, while alignment to 32B and 64B requires 16B padding. In this case, the larger of the splitting granularities that does not require zero padding can be preferred as Pci, i.e., 16B.

[0094] In another example, assuming Ci = 28B, alignment to 8B, 16B, and 32B all require padding with 4B zeros, while alignment to 64B requires padding with 36B zeros. In this case, a smaller alignment padding amount and a larger splitting granularity can be preferred as Pci, i.e., 32B.

[0095] In another example, assuming Ci = 49B, aligning to 8B requires padding with 7B zeros, while aligning to 16B, 32B, and 64B all require padding with 15B zeros. In this case, the alignment padding amount differs by only 8B, which is within an acceptable range. Therefore, the larger splitting granularity of 64B can be preferred.

[0096] Figures 6a-6c Several examples of data width dimension folding according to embodiments of this disclosure are shown. In these examples, it is also assumed that M = 64B.

[0097] like Figure 6aAs shown, when the input channel splitting granularity Pci = 16B, the W dimension needs to be folded by a factor of 4. That is, the shape of a data row is Wi*Ci = 4 × 16B. When the size of the Ci dimension exceeds 16B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 48B, its data will be split into 3 data rows. The figure uses rounded rectangles to show the data included in each of the 3 data rows; here, 3 can also be referred to as the number of split blocks in the Ci dimension.

[0098] like Figure 6b As shown, when the input channel splitting granularity Pci = 32B, the W dimension needs to be folded by a factor of 2. That is, the shape of a data row is Wi*Ci = 2 × 32B. Similarly, when the size of the Ci dimension exceeds 32B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 96B, the data thereon will be split into 3 data rows. Only a single data row is shown in the figure.

[0099] like Figure 6c As shown, when the input channel splitting granularity Pci = 64B, the W dimension needs to be folded by 1, meaning it doesn't need to be folded. In this case, the shape of a data row is Wi*Ci = 1 × 64B. Similarly, when the size of the Ci dimension exceeds 64B, the data at 1*Ci will be split into multiple data rows. For example, when Ci = 128B, the data therein will be split into two data rows. The figure only shows a single data row.

[0100] As mentioned above, in some embodiments, Figure 5 The main processing circuit 510 can determine the input feature map as multicast data and store it in the first storage circuit 530, so that the data can be transmitted to multiple scheduled slave processing circuits via broadcast during the operation. As can be seen from the width folding scheme described above, since WC is a continuous dimension, the format of the input data does not need to be divided and dimensionality transformed, and the original input data format HWC can be received directly. Therefore, the input feature map can be stored in the first storage circuit 530 in its original format (e.g., HWC).

[0101] When the input feature map is read from the first storage circuit 530 and broadcast to multiple slave processing circuits, the aforementioned alignment process can be performed. That is, during the transmission from the first storage circuit to the buffer circuit (e.g., the first buffer circuit) within the slave processing circuit, the main processing circuit 510 can control the alignment process of the Ci dimension to align to the determined input channel splitting granularity Pci, and then fold the corresponding number of Wi dimension data to form a data row, and broadcast it to the slave processing circuit with a data row as the smallest granularity.

[0102] As can be understood, in the example of the aforementioned FUCONV convolutional layer, the input feature map is the output data of the previous layer, which has been split into two segments in the Ci dimension. Therefore, the data format can be [2,hi,wi,32B] or [2,hi,wi,64B].

[0103] Figure 7 The illustration schematically shows an input feature map storage method according to some embodiments of this disclosure. As shown, the input feature map can be stored in two segments Ci, with the starting address interval between the two segments Ci_seg.stride: the size of each segment ci is 32B or 64B. For a 32B segment, the shape of a data row is Wi*Ci = 2 × 32B; for a 64B segment, the shape of a data row is Wi*Ci = 1 × 64B.

[0104] Thus, the storage format of the input feature map and the folding process via the data channel in this disclosure embodiment have been described above.

[0105] Example convolution kernel storage

[0106] Convolution computation involves multiplying and adding each input feature map with a convolution kernel of size Co, resulting in Co output feature maps. However, on-chip memory cannot always store convolution kernels and input feature maps of all sizes simultaneously. Therefore, hardware involves a series of operations that repeatedly load input feature data or weight data. Balancing the repeated loading of input feature data or weight data significantly impacts computational efficiency. In practical applications, to reduce frequent off-chip memory accesses, different reuse methods can be adopted based on the scale and characteristics of the data involved in the computation.

[0107] As described above regarding the principles of convolution operations, the results of operations along the Co dimension do not need to be accumulated. Therefore, operations on different Co dimensions can be performed relatively independently on different computational circuits. That is, different convolution kernels of Co can be assigned to different computational circuits, using the same input feature map for computation. In this case, the input feature map is reused among these computational circuits, with the number of reuses Rn = Ns, where Ns is the number of computational circuits.

[0108] In some embodiments disclosed herein, the Co value assigned to each slave processing circuit can be determined based on the Co dimension size of the output channel of the convolution kernel and the number of schedulable slave processing circuits Ns.

[0109] To simplify the scheduling of slave processing circuits, in some embodiments, allocation can be based on the size of the output channel dimension Co of the convolution kernel, with each slave processing circuit processing one Co value per round of computation. When Co does not exceed the number of schedulable slave processing circuits, Co slave processing circuits can be scheduled, each processing one Co value. For example, when Co = 8, 8 slave processing circuits can be scheduled, each processing one Co value. When Co exceeds the number of schedulable slave processing circuits, the computation can be completed in multiple rounds. Each round schedules as many slave processing circuits as possible, each processing one Co value. For example, when Co = 24, all 16 available slave processing circuits can be scheduled in the first round to process the first 16 Co values; in the second round, 8 slave processing circuits can be scheduled to process the last 8 Co values, thus completing the entire computation.

[0110] In some embodiments, the input feature map data can be further reused along the H dimension, thereby further reducing memory access. In these embodiments, considering that some storage circuits only support reading data in ascending order of address, the data in the H dimension needs to be stored in reverse order to facilitate reading the corresponding weight data along the H dimension. This will be described in detail later in conjunction with the convolution operation process.

[0111] As mentioned earlier, in some embodiments, the convolution kernel can be defined as distribution data and stored in the second storage circuit 540 for distribution to the corresponding slave processing circuit or for reading by the slave processing circuit before computation. The second storage circuit 540 can be shared by multiple (e.g., Ns) slave processing circuits 520, with each slave processing circuit allocated an independent storage area. This ensures that the data required for computation by each slave processing circuit only needs to be read from its corresponding storage area, accelerating memory access. When the convolution kernel is partitioned and stored according to the Co dimension, the convolution kernel corresponding to the Co value allocated to a slave processing circuit can be stored in the corresponding storage area of ​​the second storage circuit. Since the Co dimension is the highest storage dimension of the convolution kernel, partitioning and storing along the Co dimension does not require dimension transformation or other processing; the convolution kernel data corresponding to the Co value can be directly stored in the second storage circuit in its original format (e.g., KhKwCi).

[0112] Figure 8 A schematic diagram of a convolution kernel storage method according to an embodiment of this disclosure is shown. In this example, it is assumed that the Co dimension of the convolution kernel is 8, and thus 8 slave processing circuits are scheduled to perform operations. The figure exemplarily shows 8 storage regions 800-807 allocated for, for example, 8 slave processing circuits SL0-SL7 (Ns = 8). Each storage region stores the convolution kernel corresponding to the Co value to be processed by that slave processing circuit.

[0113] In one example, consecutive Co values ​​are sequentially assigned to eight SLs (i.e., in increments of 1). For instance, the diagram shows convolutional kernels with Co = 0 to 7 stored sequentially in eight storage regions 800 to 807. Furthermore, within each storage region, the kernels are stored in reverse order along the H direction; that is, the kernels are stored in descending order of their indices along the height dimension Kh. This facilitates reading the kernels in ascending order of their addresses when loading them onto the second buffer circuit.

[0114] Similar to the input feature map, each Co-value convolutional kernel also undergoes a similar splitting and alignment process along the Ci dimension. Likewise, in the aforementioned FUCONV convolutional layer example, the convolutional kernel has already been split into two segments along the Ci dimension, and therefore is similarly stored segmented.

[0115] In some embodiments, when the convolutional kernel is read from the second storage circuit and distributed to the corresponding slave processing circuit, Ci split alignment processing can be performed as needed. That is, during the transfer from the second storage circuit to the buffer circuit (e.g., the second buffer circuit) within the slave processing circuit, Ci-dimensional alignment processing of the convolutional kernel can be performed to align it to the previously determined input channel split granularity Pci. Unlike the input feature map, the convolutional kernel does not need to be folded in the W dimension, but is instead copied and expanded according to the folding factor Ws, as can be seen in the subsequent description of the convolution operation process.

[0116] A single exemplary convolution operation process within the processing circuit

[0117] When the input feature map is broadcast to the scheduled slave processing circuits, and the convolution kernels are distributed to the corresponding slave processing circuits, each slave processing circuit can simultaneously perform convolution operations on the corresponding data of the input feature map and the convolution kernels. Then, the main processing circuit can concatenate the operation results returned by multiple slave processing circuits according to the convolution width folding scheme to obtain the output feature map of the convolution operation between the input feature map and the convolution kernels. Specifically, multiple computational circuits (CUs) and various buffer circuits in the slave processing circuits can be utilized (see...). Figure 5 The specific convolution operation is performed by a buffer circuit. Depending on the size of the buffer circuit inside the processing circuit and the computing power limitations of the computing circuit, multiple operation cycles are usually required to complete the required operation in each round of operation.

[0118] In some embodiments, the first buffer circuit can be used to buffer the input feature map from the first storage circuit; correspondingly, the second buffer circuit can be used to buffer the convolution kernel, i.e., the weight data, from the second storage circuit. Each arithmetic circuit CU can perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit (e.g., input feature rows) and data rows selected from the second buffer circuit (e.g., partial weight rows or extended weight rows) in each arithmetic cycle. For simplicity, the following description focuses on the processing of a single Co value within a single processing circuit SL; it is understood that similar processing is performed in other SLs.

[0119] As we know from the convolution operation principle above, the value of each convolution output point on the output feature map corresponds to the sum of the positional multiplications of the input feature map and the weights within its convolution window. That is, the value of a single output point is the sum of the positional multiplications of each part.

[0120] In some embodiments, for a single output point in the output feature map, the value of the output point can be calculated in the following order and multiple loops, wherein: the Kw dimension of the convolution kernel is used as the inner loop to calculate the partial sum of the output point, and the number of loops is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel, and Kmax is the maximum convolution kernel width value supported by the processing circuit; the number of blocks Bci of the convolution kernel divided by Pci in the Ci dimension is used as the middle loop to calculate the partial sum of the output point, and the number of loops is Nci = Bci = ceil(Ci / Pci); the Kh dimension of the convolution kernel is used as the outer loop to calculate the partial sum of the output point, and the number of loops is Nkh = Kh, where Kh is the height dimension of the convolution kernel; and the sums of each part are accumulated to obtain the value of the output point, wherein the total number of loops is Ncycle = Nkw * Nci * Nkh.

[0121] Figure 9 An exemplary loop diagram illustrating the computation of a single convolutional output point according to an embodiment of this disclosure is shown. In this example, it is assumed that the convolutional kernel has Kw = 2, Ky = 3, Ci is divided into two segments, each 32 bytes; the input feature map has Wi = 20, Hi = 20, Ci is also divided into two segments, each 32 bytes; and the convolutional stride in the width and height directions is Sx = Sy = 1. The figure shows the various parts and components of the first output point on the output feature map, with each data point represented by its height and width coordinates.<h,w> This indicates that the magnitude of each data point in the ci direction is Pci.

[0122] In the inner loop of dimension Kw, the input feature rows and expanded weight rows are synchronously selected in the width dimension by sliding with a step size of 1 on the first and second buffer circuits to calculate different partial sums of the same output point. The number of sliding steps in the inner loop, i.e., the number of loops, is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel, and Kmax is the maximum convolution kernel width value supported by the processing circuit.

[0123] In some embodiments, Kmax can be determined as follows:

[0124] Kmax = L1*Ws - Ncu*Ws + 1

[0125] Where L1 is the size of the first buffer circuit, in data rows; Ncu is the number of scheduled computational circuits; and Ws is the folding factor of the width dimension. For example, with a first buffer circuit of L1 = 8 data rows and Ncu = 4, Kmax = 17 when Ws = 4; Kmax = 9 when Ws = 2; and Kmax = 5 when Ws = 1. It can be seen that in most cases, the width dimension Kw of the convolution kernel will not exceed Kmax; therefore, Nkw = Kw.

[0126] like Figure 9 As shown, in this example, the number of inner loop iterations in dimension Kw is Nkw = Kw = 2. Specifically, the first iteration selects the input feature data point <0,0> and the weight data point <0,0> and performs positional multiplication and accumulation to obtain the first partial sum; the second iteration simultaneously slides one step to the right, selects the input feature data point <0,1> and the weight data point <0,1> and performs positional multiplication and accumulation to obtain the second partial sum. It can be seen that both the first and second partial sums belong to the partial sums of the first output point <0,0>.

[0127] In the middle loop, the loop iterates through the number of segments Bci split by Pci according to the Ci dimension. Figure 9 In the example, Nci = Bci = 2. Therefore, the input feature map and weights are selected synchronously. First, data is selected from the first segment Ci_seg = 0, and positional multiplication and accumulation are performed to obtain the third part sum. Second, data is selected from the second segment Ci_seg = 1, and positional multiplication and accumulation are performed to obtain the fourth part sum. As can be seen from the principle of convolution, the product results along the Ci dimension also need to be accumulated. Therefore, both the third and fourth part sums belong to the partial sums of the first output point <0,0>. It can also be understood that the third part sum is essentially the sum of the first and second part sums obtained from the inner loop. The fourth part sum is similar.

[0128] In the outer loop of dimension Kh, the sum of each part can be calculated by looping Kh times in the H direction according to the size of Kh. As shown in the figure, if Kh = 3, three loops are required. The first loop selects weights from the row where Kh = 0 and input feature maps from the row where Hi = 0, performs positional multiplication and accumulation to obtain the fifth part sum; the second loop selects weights from the row where Kh = 1 and input feature maps from the row where Hi = 1, performs positional multiplication and accumulation to obtain the sixth part sum; the third loop selects weights from the row where Kh = 2 and input feature maps from the row where Hi = 2, performs positional multiplication and accumulation to obtain the seventh part sum. It can be seen that the fifth, sixth, and seventh part sums all belong to the part sums of the first output point <0,0>. It can also be understood that the fifth part sum is essentially the sum of the third and fourth part sums obtained from the middle loop. The sixth and seventh part sums are similar. Since the data in dimension Kh is not folded or split in any dimension, the convolution scheme of this embodiment can support convolution strides of any value in dimension Kh.

[0129] It's understandable that when the width of the convolutional kernel exceeds Kmax, it needs to be split along the Kw direction according to that maximum kernel width. In this case, in addition to the three loops mentioned above, further loop processing is performed according to the split along Kw.

[0130] As mentioned earlier, in some embodiments, the input feature map data can be further reused in the H dimension, thereby further reducing memory access. Specifically, the selected input feature rows can be reused rn times each time, and each time they are multiplied and accumulated with the rn extended weight rows corresponding to the convolution kernel in the height dimension to obtain rn consecutive output blocks of the output feature map in the height dimension, where rn is determined according to the height dimension size Kh of the convolution kernel and the convolution stride Sy in the height direction of the convolution operation.

[0131] Figure 10 This illustration shows a computational diagram of reusing input feature map data along the H dimension according to some embodiments of this disclosure. The parameter configuration of this example is... Figure 9 similar.

[0132] As shown in the figure, when the same input feature data point is multiplied and accumulated along the H dimension through Kh weight points, the resulting partial sums belong to different output points. To avoid computational overflow, let's take the input feature data point <2,0> as an example. When the input feature data point <2,0> is multiplied and accumulated with the weight data point <0,0>, it's equivalent to the case of convolution window A, resulting in the eighth partial sum, which belongs to the output point <2,0>; when the input feature data point <2,0> is multiplied and accumulated with the weight data point <1,0>, it's equivalent to the case of convolution window B, resulting in the ninth partial sum, which belongs to the output point <1,0>; when the input feature data point <2,0> is multiplied and accumulated with the weight data point <2,0>, it's equivalent to the case of convolution window C, resulting in the tenth partial sum, which belongs to the output point <0,0>.

[0133] Therefore, the number of times the input feature map is reused in the H dimension depends on the maximum overlap of adjacent convolutional windows in the H dimension. For example, in the example above, Kh = 3, Sy = 1, the input feature data point <2,0> is simultaneously covered by the three convolutional windows corresponding to the three output points (i.e., output points <2,0>, <1,0>, and <0,0>), and thus can be reused 3 times. It can be understood that when Sy > 1, the number of reuses rn is less than Kh, rn = Kh - Sy + 1; and some data points are not covered by overlapping convolutional windows, that is, they do not need to be reused.

[0134] The above describes how to obtain the value of a single output point by repeatedly calculating the partial sum in multiple loops, and how to interleave input feature map reuse in the calculation of a single output point to calculate multiple output points / output blocks in the H dimension.

[0135] To fully utilize the parallel operation of multiple computational circuits within a single slave processing circuit, the output feature map can be computed in parallel by multiple computational circuits (CUs) within a single slave processing circuit. Considering the dimensional storage order of the output feature map and the W-folding of the input feature map, to simplify output processing, preferably, Ncu output blocks are sequentially divided along the Wo dimension, so that each block is computed in parallel by Ncu computational circuits. Each output block corresponds to the computation result of one input feature data line. In some embodiments, Ncu adjacent input feature lines are sequentially selected from a first buffer circuit and distributed to Ncu computational circuits, and a corresponding extended weight data line is selected or generated from a second buffer circuit and broadcast to Ncu computational circuits, thereby achieving parallel computation of Ncu output blocks by reusing weight data.

[0136] Figure 11 A schematic splitting method of the output feature map according to an embodiment of this disclosure is shown. For simplicity, Figure 11This example shows the split of the output feature map along the Wo dimension for a single Co value. In this example, assuming Ncu = 4, the output map is divided into 4 output blocks along the Wo dimension, with each output block corresponding to the result of the computation of one row of input feature data.

[0137] Furthermore, depending on the different data formats within a data row of the input feature map, the output block computed by a single computational circuit CU can include different numbers of output points. Specifically, based on the previously determined width dimension folding factor Ws, each output block includes Ws consecutive output points along the width Wo dimension. For example, when the input feature map Ci is split at a granularity of Pci = 16B, a data row includes 4 Wi, and output points in 4 Wo dimensions can be computed; when the input feature map Ci is split at a granularity of Pci = 32B, a data row includes 2 Wi, and output points in 2 Wo dimensions can be computed; and when the input feature map Ci is split at a granularity of Pci = 64B, a data row includes 1 Wi, and output points in 1 Wo dimension can be computed. Figure 11 The diagram further illustrates the different configurations of a single output block in the three cases described above, including 4, 2, or 1 Wo output points, respectively.

[0138] To support a single CU simultaneously computing one or more Wo output points that may be included in an output block, in some embodiments, the corresponding weight data can be constructed as follows: When distributing the convolution kernel in the second storage circuit to the second buffer circuit of each slave processing circuit, in addition to aligning the Ci dimension of the convolution kernel to Pci, a column of Ci data split or aligned to Pci in the Ci dimension is copied and expanded into Ws columns according to the folding factor Ws in the width dimension, forming an extended weight data row, which is stored in the second buffer circuit. That is, the shape of an extended weight data row is Ws*Pci, which can correspond to an input feature data row. Thus, an extended weight data row can be selected from the second buffer circuit and broadcast to N in the slave processing circuit. CU Each arithmetic circuit can then perform bitwise multiplication and accumulation on a data line of Pci / M = 1 / Ws, for an input feature line from the first buffer circuit and an extended weight data line from the second buffer circuit, to obtain a partial sum of M / Pci = Ws output points.

[0139] In other embodiments, the copying and expansion process of the weight data described above can also be performed on the data path from the second buffer circuit to the arithmetic circuit, and the processing method is similar, so it will not be described in detail here.

[0140] Therefore, the above calculation process involves two layers of weight reuse: the first layer is between the computational circuits (CUs), where the weights are broadcast to Ncu computational circuits, resulting in Ncu reuse times; the second layer is between one or more Wo output points within each computational circuit, where the weights are expanded to calculate Ws output points within each CU, resulting in Ws reuse times. Thus, by reusing data as much as possible, frequent data access and memory access can be effectively reduced.

[0141] It can also be understood that when the size of the Wo dimension of the output feature map exceeds the computation cost in a single operation, for example, Wo > Ws * Ncu, then Wo can be processed in a loop according to the Ws * Ncu split.

[0142] In some embodiments, for the output feature map on a single output channel Co, the output points thereon can be calculated by splitting it as follows: the output feature map is split into blocks of size (Ws*Ncu)*Ho along the width dimension, and the output points are calculated for each block, where Ncu is the number of schedulable arithmetic circuits in the processing circuit, and Ho is the height dimension of the output feature map; for each block, the output circuits are calculated in the order of width dimension first, then height dimension.

[0143] When the processing circuit writes the calculation results from the arithmetic circuit, it can store the calculation results of each arithmetic circuit in, for example, the order of Wo dimension first, then Ho dimension. Figure 5 In the third buffer circuit, when a slave processing circuit outputs the output points of its internal arithmetic circuits, it can output the output points calculated by multiple arithmetic circuits in a specific order according to the division of output points, facilitating subsequent processing. For example, each slave processing circuit processes convolution kernels for different output channel Co values, and can output the calculation results of each arithmetic circuit in turn, following the order of width dimension Wo first, then height dimension Ho. Correspondingly, the main processing circuit in the computing device can concatenate and store the calculation results output from each slave processing circuit according to the order of Co values, in the dimensional storage order of HoWoCo.

[0144] As can be seen from the preceding calculation process, each time Ncu*Ws output points are calculated in the Wo dimension of the output feature map. That is, the output points are aligned to Ncu*Ws. Therefore, there may be redundant output points calculated. These redundant output points in the Wo dimension can be filtered out in the data path storing the calculation results.

[0145] The following describes the detailed operation process of convolution in the embodiments disclosed herein with reference to specific examples.

[0146] Example: Ci is divided into two segments, with Bci = 2 segments, each segment being 32 bytes, and Co = 8 bytes.

[0147] Figures 12a-12cThe diagram illustrates the computation process of a convolution operation scheme according to an embodiment of this disclosure. In this embodiment, Ci is split into two segments, Ci_seg = 0 to 1, each segment being 32 bytes. Therefore, the format of an input feature data row is 2 × 32 bytes (WiCi). The diagram shows that each row of data includes 2 columns of Wi data, so the output block calculated by one computation circuit includes 1 × 2 (CoWo) output points. Co = 8, so only Ns = 8 slave processing circuits need to be scheduled, and each slave processing circuit processes 1 Co value. It is not unreasonable to assume that the size of the convolution kernel is KhKw = 3 × 2. In the following description, height and width dimension coordinates are used.<h,w> Let represent each data point, and each data point has a size of Pci in the Ci dimension, which is 32B in this example.

[0148] Figure 12a The calculation process of the middle loop of Ci_seg and the inner loop of the Kw dimension is shown for hi=0. N is selected from the first buffer circuit according to the method corresponding to the division of the output block. CU Each of the N input feature lines is sent to N. CU Each arithmetic circuit selects an extended weight row from the second buffer circuit and broadcasts it to the Ncu arithmetic circuits for calculation.

[0149] During the first calculation period indicated by arrow ①, numbers are selected from the data segment where Ci_seg = 0. Specifically, a data row consisting of input feature points <0,0> and <0,1> is selected and sent to the first operational circuit CU0; a data row consisting of input feature points <0,2> and <0,3> is selected and sent to operational circuit CU1; a data row consisting of input feature points <0,4> and <0,5> is selected and sent to operational circuit CU2; and a data row consisting of input feature points <0,6> and <0,7> is selected and sent to operational circuit CU3 (the number selection is shown in the figure with a black dashed box). Correspondingly, the extended weight row A0A0, which is expanded from the data point <0,0> (hereinafter referred to as "A0") in the convolution kernel data segment where Ci_seg = 0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively to obtain the partial sum of the eight output points w0 to w7 at ho = 0, and each operational circuit calculates the two adjacent output points.

[0150] Since the data in the row with hi=0 is not reused in the H dimension, there is no need to reuse it in the H dimension at this point. Therefore, the inner loop in the Kw dimension can continue.

[0151] During the second calculation period indicated by arrow ②, numbers are still selected from the data segment Ci_seg=0, but a sliding operation along the W dimension is required. At this time, the first buffer circuit slides one step in the Wi direction to select the corresponding four input feature rows (shown as small gray dashed boxes in the diagram), which are then sent to the four computation circuits respectively; and the second buffer circuit slides one step in the Kw direction to select the extended weight row B0B0, which is expanded from the data point <0,1> (hereinafter referred to as "B0"), and broadcasts it to the four computation circuits. Thus, the four computation circuits perform bitwise multiplication and accumulation operations respectively. Since the input feature map and weights slide synchronously, the result is still a partial sum of the eight output points w0 to w7 at ho=0, which is accumulated on the partial sum of the previous calculation.

[0152] At this point, the inner loop in the Kw dimension ends, meaning the partial sums in the Kw direction have been calculated. Next, the middle loop in the Ci_seg dimension begins. The above number selection and calculation process is repeated for the data segment where Ci_seg = 1.

[0153] During the third calculation period indicated by arrow ③, numbers are selected from the convolution kernel data segment (Ci_seg=1) and the input feature map data segment, respectively. Specifically, a data row consisting of input feature points <0,0> and <0,1> from the input feature map data segment (Ci_seg=1) is selected and sent to the first operation circuit CU0; a data row consisting of input feature points <0,2> and <0,3> is selected and sent to operation circuit CU1; a data row consisting of input feature points <0,4> and <0,5> is selected and sent to operation circuit CU2; and a data row consisting of input feature points <0,6> and <0,7> is selected and sent to operation circuit CU3 (the number selection is shown in the figure with a black dashed box). Correspondingly, an extended weight row a0a0, formed by expanding data point <0,0> (hereinafter referred to as "a0") from the convolution kernel data segment (Ci_seg=1), is selected and broadcast to the four operation circuits. Thus, the four operation circuits perform bitwise multiplication and accumulation operations respectively. Since the input feature map and weights are taken synchronously in the Ci dimension, according to the principle of convolution operation, what is obtained is still the partial sum of the 8 output points w0 to w7 in the row where ho = 0. These partial sums are accumulated on the partial sum calculated in the previous time.

[0154] During the fourth calculation period indicated by arrow ④, numbers are still selected from the data segment Ci_seg=1, but a sliding operation along the W dimension is required. At this time, the first buffer circuit slides one step in the Wi direction to select the corresponding four input feature rows (shown as small gray dashed boxes in the diagram), which are then sent to the four computation circuits. Additionally, the second buffer circuit slides one step in the Kw direction to select the extended weight row b0b0, which is expanded from the data point <0,1> (hereinafter referred to as "b0"), and broadcasts it to the four computation circuits. Thus, the four computation circuits perform bitwise multiplication and accumulation operations respectively. Since the input feature map and weights slide synchronously, the result is still the partial sum of the eight output points w0 to w7 in the row ho=0, which is accumulated on the partial sum of the previous calculation.

[0155] This completes the middle loop of Ci_seg and the inner loop of Kw when hi=0.

[0156] Next, we can perform an outer loop, which means adding 1 to the H dimension.

[0157] Figure 12b The loop processing for hi=1 is shown. At this point, the first buffer circuit stores the data for the row where hi=1. Figure 12a Similarly, firstly, following the method corresponding to the output block partitioning, four input feature rows are selected from the first buffer circuit and sent to the four arithmetic circuits respectively. Then, one extended weight row is selected from the second buffer circuit and broadcast to the four arithmetic circuits for calculation. Figure 12a The difference is that the data in the row with hi=1 is reused in the H dimension. That is, this row of data can be used to calculate the data points with ho=0 in the output feature map, and it can also be used to calculate the data points with ho=1 in the output feature map, so it can be reused twice.

[0158] Specifically, during the first calculation period indicated by arrow ①, numbers are selected from the data segment Ci_seg=0. Four data rows, shown in the black dashed box in the diagram, are chosen and sent to the four computational circuits. At this point, multiplexing along the H dimension is applied. To calculate the output points along the H dimension sequentially, the weight data needs to be extracted in reverse order along the H dimension. First, the extended weight row A1A1, formed by expanding the data point <1,0> (hereinafter referred to as "A1") from the convolution kernel data segment of Ci_seg=0, is selected and broadcast to the four computational circuits. The four computational circuits then perform bitwise multiplication and accumulation operations to obtain the partial sums of the eight output points w0 to w7 in the row ho=0. These partial sums are then added to the previously calculated partial sums of the corresponding output points.

[0159] Next, during the second calculation period indicated by arrow ②, keeping the input feature rows of each operational circuit unchanged, the extended weight row A0A0, which is formed by expanding the data point <0,0> (i.e., "A0") in the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=1, with each operational circuit calculating the sum of two adjacent output points.

[0160] At this point, the reuse of the input feature map along the H dimension is complete. The next iteration along the Kw dimension then begins.

[0161] During the third calculation period indicated by arrow ③, numbers are still selected from the data segment where Ci_seg=0, but this time they slide one step in the W dimension. At this point, the corresponding four input feature rows are selected by sliding one step in the Wi direction from the first buffer circuit (for clarity, the data in the first buffer circuit is repeated in the figure, and the selected numbers are indicated by a slightly smaller gray dashed box), and sent to the four computation circuits respectively. Similarly, multiplexing is also inserted in the H dimension. First, the extended weight row B1B1, which is expanded from the data point <1,1> (hereinafter referred to as "B1") in the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four computation circuits. Thus, the four computation circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=0, and adding it to the previous result.

[0162] Next, during the fourth calculation period indicated by arrow ④, while keeping the input feature rows of each operational circuit unchanged, the extended weight row B0B0, which is formed by expanding the data point <0,1> (i.e., "B0") in the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=1, and adding it to the previous result.

[0163] At this point, the inner loop in the Kw dimension ends, meaning the partial sums in the Kw direction have all been calculated. Next, the middle loop in the Ci_seg dimension begins. The above number selection and calculation process is repeated for the data segment where Ci_seg = 1, also embedding reuse in the H dimension, for a total of 4 calculations, which will not be detailed here. For simplicity, only the inner loop process is shown in the diagram; the calculation process of the middle loop can be derived similarly.

[0164] This completes the middle loop of Ci_seg and the inner loop of Kw when hi=1.

[0165] Next, we can continue the outer loop, that is, add 1 to the H dimension, hi = 2.

[0166] Figure 12c The loop processing for hi=2 is illustrated. At this point, the first buffer circuit stores the data for the row where hi=2. Similarly, following the division method corresponding to the output block, four input feature rows are first selected from the first buffer circuit and sent to four computational circuits respectively. An extended weight row is selected from the second buffer circuit and broadcast to the four computational circuits for calculation. The data for the row hi=2 is reused in the H dimension, and this row can be used to calculate the data points ho=0, ho=1, and ho=2 in the output feature map, allowing for up to three reuses.

[0167] Specifically, during the first calculation period indicated by arrow ①, numbers are selected from the data segment Ci_seg=0. Four data rows, shown in the black dashed box in the diagram, are chosen and sent to the four computational circuits. At this point, multiplexing along the H dimension is applied. To calculate the output points along the H dimension sequentially, the weight data needs to be extracted in reverse order along the H dimension. First, the extended weight row A2A2, formed by expanding the data point <2,0> (hereinafter referred to as "A2") from the convolution kernel data segment of Ci_seg=0, is selected and broadcast to the four computational circuits. The four computational circuits then perform bitwise multiplication and accumulation operations to obtain the partial sums of the eight output points w0 to w7 in the row ho=0. These partial sums are then added to the previously calculated partial sums of the corresponding output points.

[0168] Next, during the second calculation period indicated by arrow ②, keeping the input feature rows of each operational circuit unchanged, the extended weight row A1A1, which is formed by expanding the data point <1,0> (i.e., "A1") in the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sums of the eight output points w0 to w7 in the row where ho=1. These partial sums are then added to the previously calculated partial sums of the corresponding output points.

[0169] Next, during the third calculation period indicated by arrow ③, the input feature rows of each operational circuit remain unchanged. The extended weight row A0A0, formed by expanding the data point <0,0> (i.e., "A0") within the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=2.

[0170] At this point, the reuse of the input feature map along the H dimension is complete. The next iteration along the Kw dimension then begins.

[0171] During the fourth calculation period indicated by arrow ④, numbers are still selected from the data segment where Ci_seg = 0, but this time they slide one step in the W dimension. At this point, the corresponding four input feature rows are selected by sliding one step in the Wi direction from the first buffer circuit (for clarity, the data in the first buffer circuit is repeated in the figure, and the selected numbers are indicated by a slightly smaller gray dashed box), and sent to the four computation circuits respectively. Similarly, multiplexing is also inserted in the H dimension. First, the extended weight row B2B2, which is expanded from the data point <2,1> (hereinafter referred to as "B2") in the convolution kernel data segment where Ci_seg = 0, is selected and broadcast to the four computation circuits. Thus, the four computation circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho = 0, and adding it to the previous result.

[0172] Next, during the fifth calculation period indicated by arrow ⑤, keeping the input feature rows of each operational circuit unchanged, the extended weight row B1B1, which is formed by expanding the data point <1,1> (i.e., "B1") in the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=1, and adding it to the previous result.

[0173] Next, during the sixth calculation period indicated by arrow ⑥, the input feature rows of each operational circuit remain unchanged. The extended weight row B0B0, formed by expanding data points <0,1> (i.e., "B0") from the convolution kernel data segment where Ci_seg=0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sum of the eight output points w0 to w7 in the row where ho=2, and adding it to the previous result.

[0174] At this point, the inner loop in the Kw dimension ends, meaning the partial sums in the Kw direction have all been calculated. Next, the middle loop in the Ci_seg dimension begins. The above number selection and calculation process is repeated from the data segment where Ci_seg = 1, also embedding reuse in the H dimension, for a total of 6 calculations, which will not be detailed here. For simplicity, only the inner loop process is shown in the diagram; the calculation process of the middle loop can be derived similarly.

[0175] This completes the middle loop of Ci_seg and the inner loop of Kw when hi=2. At this point, the values ​​of the eight output points w0 to w7 in the row where ho=0 on the output feature map have also been accumulated and can be output.

[0176] Next, the outer loop can continue, that is, adding 1 to the H dimension, hi = 3. Repeat this loop until the entire H dimension has been processed.

[0177] After the outer H-dimensional loop is also processed, each computational circuit can accumulate to obtain a final convolution result of ho*Ws four output points. One slave processing circuit with four computational circuits obtains ho*(Ws*4) output points on the same Co. Eight slave processing circuits obtain a total of eight ho*(Ws*4) output points on Co.

[0178] Figure 13 A schematic diagram illustrating the logic for writing and outputting computation results according to an embodiment of this disclosure is shown.

[0179] As shown in the figure, multiple arithmetic circuits (CUs) within a single slave processing circuit (SL) can sequentially write the computation results into a result buffer circuit (e.g., according to the computation order). Figure 5 In the third buffer circuit. Specifically, the output points of the same Co calculated by each CU can be written first in the order of Wo (writing loop ①). Then, the output points of different Ho calculated by each CU can be written in the order of Ho (writing loop ②). For example, for SL0, first write w0 to w7 when ho = 0, then write w0 to w7 when ho = 1, then w0 to w7 when ho = 2, and so on. Similar result writing is performed in other SLs, except that the Co values ​​are processed differently.

[0180] The read order can be the same as the write order, starting with the Wo dimension and then moving to the Ho dimension. More specifically, the results can be read sequentially from the result buffers of each processing circuit in the Co order, while reading the results from each CU in the Wo order. For example, first read the two output points w0 and w1 calculated by each CU0 in the eight SLs, then the two output points w2 and w3 calculated by each CU1, then w4 and w5 calculated by CU2, and finally w6 and w7 calculated by CU3 (read loop ①). Then, read the output points on each Ho in the Ho order (read loop ②). Figure 13 The right-hand view shows the readout results. Note that when reading in Co order, the results are read in turn on the 8 SL result buffer circuits so that the Co dimensions are continuous, for example from 0 to 7.

[0181] The specific convolution operation process described above in conjunction with the embodiments illustrates the convolution optimization scheme provided by the embodiments of this disclosure. It is understood that, depending on the different values ​​of Ci and Co, there can be many more combinations to obtain different embodiments. Furthermore, based on the teachings of this disclosure, those skilled in the art can conceive of other convolution optimization schemes according to specific hardware circuit configurations (such as the number of processing circuits, the number of operational circuits within the processing circuits, the hardware's single-processing capability, etc.), all of which fall within the scope of this disclosure and will not be enumerated here.

[0182] This disclosure also provides a chip that may include the data processing apparatus of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a circuit board that may include the aforementioned chip.

[0183] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0184] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0185] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0186] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0187] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0188] The foregoing can be better understood in accordance with the following terms:

[0189] Clause 1. A computing device comprising a plurality of slave processing circuits, each slave processing circuit comprising a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:

[0190] The first buffer circuit is used to buffer multiple input feature lines that will be used to perform convolution operations. One of the input feature lines includes the amount of data Pci×Ws=M in the input feature map, where Pci is the splitting granularity of the input channel Ci dimension, Ws is the folding factor of the width W dimension, and M is the amount of data processed by the hardware in a single operation.

[0191] The second buffer circuit is used to cache the weight data for the convolution operation to be performed; and

[0192] Each of the aforementioned computational circuits is configured to perform a bitwise multiplication and accumulation operation on each computation for an input feature row selected from the first buffer circuit and an extended weight row selected or generated from the second buffer circuit, respectively, wherein one of the extended weight rows is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolutional kernel into a Ws column.

[0193] Clause 2. The computing device according to Clause 1, wherein each of the said arithmetic circuits is further configured to:

[0194] The selected input feature rows are reused rn times, and each is multiplied and accumulated with the rn extended weight rows corresponding to the convolution kernel in the height dimension to obtain rn consecutive output blocks of the output feature map in the height dimension, where rn is determined according to the height dimension size Kh of the convolution kernel and the convolution stride Sy in the height direction of the convolution operation.

[0195] Clause 3. The computing device according to Clause 2 further includes a weight storage circuit for storing the convolution kernels, wherein the convolution kernels are stored in descending order of index in the height dimension so that they can be read in ascending order of address when loaded into the second buffer circuit.

[0196] Clause 4. A computing device according to any one of Clauses 2-3, wherein for a single output point in an output feature map, the arithmetic circuit calculates the value of the output point in a multi-level cyclical manner in the following order:

[0197] The Kw dimension of the convolution kernel is used as the inner loop to calculate the partial sum of the output points, and the number of loops is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel and Kmax is the maximum convolution kernel width value supported by the processing circuit.

[0198] The number of blocks Bci split along the Ci dimension of the convolution kernel according to Pci is used as the partial sum of the output points in the middle layer loop, and the number of loops Nci = Bci = ceil(Ci / Pci);

[0199] The Kh dimension of the convolution kernel is used as the partial sum of the output points in the outer loop, with the number of loops Nkh = Kh, where Kh is the height dimension of the convolution kernel; and

[0200] The sum of each part is accumulated to obtain the value of the output point, where the total number of cycles Ncycle = Nkw * Nci * Nkh.

[0201] Clause 5. The computing device according to Clause 4, wherein in the inner loop, each of the slave processing circuits is further configured to:

[0202] From the first buffer circuit and the second buffer circuit, the input feature row and the extended weight row are synchronously selected in the width dimension to calculate the different parts of the same output point, and the number of selections is Nkw.

[0203] Clause 6. The computing device according to Clause 5, wherein in each sliding selection calculation, each of the arithmetic circuits performs the multiplexing rn times for a selected input feature row.

[0204] Clause 7. The computing device according to any one of Clauses 2-6, wherein for an output characteristic map on a single output channel Co, each of the slave processing circuits calculates its output point as follows:

[0205] The output feature map is divided into blocks of size (Ws*Ncu)*Ho according to the width dimension, and the output points are calculated for each block. Here, Ncu is the number of schedulable computing circuits in the processing circuit, and Ho is the height dimension of the output feature map.

[0206] For each block, the output circuit is calculated in the order of width dimension first, followed by height dimension.

[0207] Clause 8. The computing device according to Clause 7, wherein for each block, each of the processing circuits calculates the output point in the width dimension as follows:

[0208] It utilizes its internally schedulable Ncu arithmetic circuits to compute in parallel Ncu output blocks that are consecutive in the width dimension of the output feature map, each output block comprising Ws output points that are consecutive in the width dimension.

[0209] Clause 9. The computing device according to Clause 8, wherein each of the said slave processing circuits is further configured to:

[0210] Ncu adjacent input feature rows are selected from the first buffer circuit and distributed to the Ncu arithmetic circuits for calculation;

[0211] Select or generate a corresponding extended weight row from the second buffer circuit and broadcast it to the Ncu arithmetic circuits;

[0212] At the Ncu arithmetic circuits, bitwise multiplication and accumulation are performed on the distributed input feature lines and the broadcast extended weight lines in units of 1 / Ws data lines to obtain the partial sum of Ws output points.

[0213] Clause 10. A computing device according to any one of Clauses 7-9, wherein for each block, each of the said slave processing circuits calculates its output points in the height dimension as follows:

[0214] At each arithmetic circuit, by multiplexing rn input feature rows, the partial sum of rn consecutive output blocks in the height dimension of the output feature map is calculated sequentially, and each output block includes Ws consecutive output points in the width dimension.

[0215] Clause 11. A computing device according to any one of Clauses 1-10, wherein:

[0216] Each of the aforementioned processing circuits processes convolution kernels for different output channels co, and outputs the computation results of each processing circuit in turn, first in the width dimension Wo, then in the height dimension Ho; and

[0217] The computing device is further configured to: concatenate and store the computation results output from each processing circuit according to the order of HoWoCo dimension storage, in the order of the co values.

[0218] Clause 12. A chip comprising a computing device according to any one of Clauses 1-11.

[0219] Clause 13. A board including the chip described in Clause 12.

[0220] Clause 14. A method for performing convolution operations using any of the computing devices described in Clauses 1-11.

[0221] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A computing device comprising a plurality of slave processing circuits, each slave processing circuit comprising a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein: The first buffer circuit is used to buffer multiple input feature lines that will be used to perform convolution operations. One of the input feature lines includes the amount of data Pci×Ws=M in the input feature map, where Pci is the splitting granularity of the input channel Ci dimension, Ws is the folding factor of the width W dimension, and M is the amount of data processed by the hardware in a single operation. The second buffer circuit is used to cache the weight data for the convolution operation to be performed; as well as Each of the aforementioned computational circuits is configured to perform a bitwise multiplication and accumulation operation on each computation for an input feature row selected from the first buffer circuit and an extended weight row selected or generated from the second buffer circuit, respectively, wherein one of the extended weight rows is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolutional kernel into a Ws column.

2. The computing device according to claim 1, wherein, Each of the aforementioned operational circuits is further used for: The selected input feature rows are reused rn times, and each is multiplied and accumulated with the rn extended weight rows corresponding to the convolution kernel in the height dimension to obtain rn consecutive output blocks of the output feature map in the height dimension, where rn is determined according to the height dimension size Kh of the convolution kernel and the convolution stride Sy in the height direction of the convolution operation.

3. The computing device according to claim 2 further includes a weight storage circuit for storing the convolution kernel, wherein the convolution kernel is stored in descending order of index in the height dimension so that it can be read in ascending order of address when loaded into the second buffer circuit.

4. The computing device according to any one of claims 2-3, wherein for a single output point in the output feature map, the arithmetic circuit calculates the value of the output point in a multi-level cyclical manner in the following order: The Kw dimension of the convolution kernel is used as the inner loop to calculate the partial sum of the output points, and the number of loops is Nkw = min(Kw, Kmax), where Kw is the width dimension of the convolution kernel and Kmax is the maximum convolution kernel width value supported by the processing circuit. The number of blocks Bci split along the Ci dimension of the convolution kernel according to Pci is used as the partial sum of the output points in the middle layer loop, and the number of loops Nci = Bci = ceil(Ci / Pci); The Kh dimension of the convolution kernel is used as the partial sum of the output points in the outer loop, with the number of loops Nkh = Kh, where Kh is the height dimension of the convolution kernel; and The sum of each part is accumulated to obtain the value of the output point, where the total number of cycles Ncycle = Nkw * Nci * Nkh.

5. The computing device of claim 4, wherein in the inner loop, each of the slave processing circuits is further configured to: From the first buffer circuit and the second buffer circuit, the input feature row and the extended weight row are synchronously selected in the width dimension to calculate the different parts of the same output point, and the number of selections is Nkw.

6. The computing device of claim 5, wherein in each sliding selection calculation, each of the arithmetic circuits performs the multiplexing rn times for a selected input feature row.

7. The computing device according to any one of claims 2-6, wherein for an output feature map on a single output channel Co, each of the slave processing circuits calculates its output point as follows: The output feature map is divided into blocks of size (Ws*Ncu)*Ho according to the width dimension, and the output points are calculated for each block. Here, Ncu is the number of schedulable computing circuits in the processing circuit, and Ho is the height dimension of the output feature map. For each block, the output circuit is calculated in the order of width dimension first, followed by height dimension.

8. The computing device of claim 7, wherein for each block, each of the slave processing circuits calculates the output point in the width dimension as follows: It utilizes its internally schedulable Ncu arithmetic circuits to compute in parallel Ncu output blocks that are consecutive in the width dimension of the output feature map, each output block comprising Ws output points that are consecutive in the width dimension.

9. The computing device of claim 8, wherein each of the slave processing circuits is further configured to: Ncu adjacent input feature rows are selected from the first buffer circuit and distributed to the Ncu arithmetic circuits for calculation; Select or generate a corresponding extended weight row from the second buffer circuit and broadcast it to the Ncu arithmetic circuits; At the Ncu arithmetic circuits, bitwise multiplication and accumulation are performed on the distributed input feature lines and the broadcast extended weight lines in units of 1 / Ws data lines to obtain the partial sum of Ws output points.

10. The computing device according to any one of claims 7-9, wherein for each block, each of the slave processing circuits calculates its output points in the height dimension as follows: At each arithmetic circuit, by multiplexing rn input feature rows, the partial sum of rn consecutive output blocks in the height dimension of the output feature map is calculated sequentially, and each output block includes Ws consecutive output points in the width dimension.

11. The computing device according to any one of claims 1-10, wherein: Each of the aforementioned processing circuits processes convolution kernels for different output channels co, and outputs the computation results of each processing circuit in turn, first in the width dimension Wo, then in the height dimension Ho; and The computing device is further configured to: concatenate and store the computation results output from each processing circuit according to the order of HoWoCo dimension storage, in the order of the co values.

12. A chip comprising a computing device according to any one of claims 1-11.

13. A circuit board comprising the chip according to claim 12.

14. A method for performing convolution operations using a computing device according to any one of claims 1-11.

Images