Computing devices, methods, and related products that perform convolution operations

Abstract

A computing device, a method for performing convolution operation by using the computing device, and related products. The computing device can be included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure optimizes the convolution operation, improves the data multiplexing efficiency, and thus improves the operation processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a computing device, a method for performing convolution operations using the computing device, a chip, and a circuit board. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Neural networks are among the most critical technologies in artificial intelligence and deep learning, with Convolutional Neural Networks (CNNs) being the most important type. The most crucial computation in CNNs is the convolution operation within the convolutional layers (Conv layers). The function of convolutional layers is to extract features from input data. Through multiple convolutions, complex features can be extracted to ensure the network has sufficient expressive and generalization capabilities. Neural network models contain numerous convolutional operations of various types, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. When neural network models are applied to different fields, such as speech recognition, machine translation, and image processing, the corresponding input feature maps and the dimensions of each weight may vary. Furthermore, different neural network models may employ different convolutional layers with unique properties and requirements. For example, U-net is a fully convolutional neural network composed entirely of convolutional layers, with a symmetrical structure and requirements such as the data input / output channel dimensions being aligned to 16 numbers. To fully leverage the hardware advantages of deep learning processors, optimizations are needed for convolution operations of different scales and / or types to improve the computational performance of executing neural network models. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes a computing device, chip, board, and corresponding method in several aspects. By segmenting the input channel dimension of the input feature map and further splitting other dimensions based on the segmentation, data of various dimensional sizes can be adapted to the hardware of convolution operations, fully utilizing hardware performance and thus improving the computational efficiency of convolution operations. Furthermore, the output channel dimension of the output feature map can also be segmented, allowing it to be directly used as the input feature map for the next convolutional layer without additional slewing. The convolution operations in the embodiments of this disclosure can be operations in various neural network models applicable to various fields, such as image processing, speech processing, text processing, etc., including but not limited to recognition and classification. The convolution operations in the embodiments of this disclosure are particularly suitable for the U-net neural network model.

[0005] In a first aspect, embodiments of this disclosure provide a computing device configured to perform convolution operations, the computing device comprising: an input storage circuit storing an input feature map and a convolution kernel, wherein the input feature map is segmented according to a splitting granularity Pci of the input channel Ci dimension; a plurality of slave processing circuits for performing convolution operations on the broadcast-transmitted input feature map and corresponding convolution kernels assigned to the slave processing circuits, respectively; and an output storage circuit for storing output feature maps output by the plurality of slave processing circuits, wherein the output feature maps are segmented according to a splitting granularity Pco of the output channel Co dimension.

[0006] In a second aspect, embodiments of this disclosure provide a chip that includes the computing device described in the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip described in the second aspect.

[0008] In a fourth aspect, embodiments of this disclosure provide a method for performing convolution operations using the computing device described in the first aspect.

[0009] Based on the computing device, chip, board, and method for performing convolution operations by the computing device provided above, the scheme of this disclosure applies different channel segmentation and dimension splitting schemes for input feature maps of different dimensions to adapt to the processing capabilities of the hardware computing device, thereby fully utilizing the parallel processing capabilities of multiple processing circuits and effectively improving the computational efficiency of convolution operations. Furthermore, the output channel dimensions of the output feature map can also be segmented, which allows it to be easily passed to the next convolutional layer as its input feature map without additional slewing. Other advantages and effects will become readily apparent from the detailed description following in conjunction with the accompanying drawings. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0012] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0013] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0014] Figure 4 This illustrates an exemplary example of the convolution operation principle that can be applied to the embodiments disclosed herein;

[0015] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0016] Figure 6 The illustration shows a schematic storage method of an input feature map according to some embodiments of this disclosure;

[0017] Figure 7 A schematic diagram of the convolution kernel storage method according to an embodiment of this disclosure is shown;

[0018] Figure 8 Several exemplary methods for dividing output blocks among multiple arithmetic circuits of a single slave processing circuit are shown according to embodiments of this disclosure;

[0019] Figure 9 This diagram illustrates a computational schematic of reusing input feature map data along the H dimension according to some embodiments of this disclosure;

[0020] Figures 10a-10d A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 1 of this disclosure is shown;

[0021] Figure 11 A schematic diagram of the writing and output logic of the calculation results according to Embodiment 1 of this disclosure is shown;

[0022] Figure 12 A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 2 of this disclosure is shown; and

[0023] Figure 13 A schematic diagram illustrating the operation process of the convolution operation scheme according to Embodiment 3 of this disclosure is shown. Detailed Implementation

[0024] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0026] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0028] Exemplary hardware environment

[0029] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0030] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0031] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0032] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0033] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0034] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0035] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0036] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0037] Figure 3 The diagram shows the internal structure of the processing core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0038] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0039] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0040] The storage module 33 is used to store or move relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. The NRAM 331 stores the input neurons, output neurons, and intermediate results after computation; the WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; the DMA 333 is connected to the DRAM 204 via bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204.

[0041] Exemplary convolution operation types

[0042] Based on the aforementioned hardware environment, in one aspect, this disclosure provides a computing device configured to perform convolution operations, thereby optimizing convolution operations, for example, in neural network models. Convolutional layers in a neural network model can perform convolution operations by applying convolution kernels (also called filters, weights, etc.) to an input feature map (also called input data, neurons, or input neurons) to extract features. A convolutional layer may contain multiple convolution kernels, each element of which corresponds to a weight coefficient and a bias.

[0043] Neural network models may contain various convolutional layers, such as convolutional layers that perform forward, regular 3D convolutional operations, and deconvolutional layers that perform depthwise convolutional operations. In reverse training, it may be necessary to perform inverse depthwise convolutional operations or cross-product convolutional operations. The embodiments disclosed herein are primarily optimized for regular 3D convolutional operations, but can also be applied to other types of convolutional operations without conflict.

[0044] In conventional 3D convolution operations, assuming the input feature map tensor shape in the convolutional layer is represented by X[N Hi Wi Ci], the tensor shape of the convolutional kernel is represented by K[Co Kh Kw Ci], and the output is Y[N HoWo Co], then the simplified mathematical formula for convolution operations can be expressed as follows:

[0045] Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0046] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, and Sh and Sw are the stride in the length and width directions. The formula ignores the bias, padding, and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N and C dimensions. The forward computation of the neural network model is independent in the N dimension, but fully connected in the C dimension. When the convolution kernel works, it scans the input features with a certain stride, performs matrix element-wise multiplication and sums the bias within the convolution window. In conventional 3D convolution operations, the results of the alignment multiplications in the H, W, and Ci directions are accumulated, hence the name 3D convolution. However, this 3D convolution has a constraint: the Ci dimension of the convolution kernel is equal to the Ci dimension of the input feature map, so the convolution kernel does not slide in the Ci direction, making it a pseudo-3D convolution. For simplicity, the above convolution operation is referred to as a regular 3D convolution operation.

[0047] Figure 4 An example of a conventional 3D convolution operation principle that can be applied to embodiments of this disclosure is shown.

[0048] The figure exemplifies a four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N Hi×Wi×Ci 3D rectangles 410. The figure also exemplifies a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co Kh×Kw×Ci 3D convolution kernels 420. The convolution result of the input data X and the convolution kernel K yields the output data Y, which is a four-dimensional data of size [N Ho Wo Co], which can be represented as N Ho×Wo×Co 3D rectangles 430.

[0049] The figure also shows a specific example of a convolution operation, where the input data is a 6×6×3 input feature map 440, omitting the N dimension; the convolution kernel is a 3×3×3 stereo convolution kernel 450, targeting a single Co; and the output data is a 4×4 output feature map 460. The specific operation process is as follows:

[0050] The convolution kernel 450 scans across the input feature map 440a with a certain stride, performs element-wise matrix multiplication on the input features within the convolution window 470, and then sums the bias values. That is, the value at each position in the output feature map 460 is obtained by performing a two-dimensional convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) in the output feature map 460 (i.e., the convolution output point) is obtained by performing a two-dimensional convolution operation between the convolution window 470 (framed by the black cube in the input feature map) and the 3D convolution kernel 450, resulting in three values, which are then summed to obtain the final value.

[0051] To obtain outputs at other locations, the position of the convolution kernel 450 can be moved on the input feature map 440, which means moving the convolution window of the output point. In the example in the figure, the convolution stride (Sw, Sh) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0,1) or (1,0) on the output feature map 460a can be obtained respectively.

[0052] As described above, a convolutional layer in a neural network contains N sets of input feature maps, each containing Hi × Wi × Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output feature map contains Ho × Wo × Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation also involves the convolutional stride (Sw, Sh), which affects the size of the output feature map.

[0053] In this paper, the input feature map, input data, neuron, or input neuron can be used interchangeably; the convolution kernel, filter, or weights can be used interchangeably; and the output feature map, output data, or output neuron can be used interchangeably. Furthermore, the H (height) and Y dimensions can be used interchangeably, as can the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In the embodiments disclosed herein, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window.

[0054] Exemplary computing device

[0055] In this disclosed embodiment, a master-slave computing device can be used to perform the above-described convolution operation. Furthermore, different data paths can be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0056] Figure 5 A schematic structural block diagram of a computing device 500 according to an embodiment of this disclosure is shown. It can be understood that this structure can be considered as... Figure 3 The detailed internal structure of the processing module of a single processing core can also be viewed as a refinement of multiple processing cores. Figure 3 The diagram shows a functional partitioning based on the processing core's computational modules. Figure 5 As shown, the computing device 500 of this disclosure embodiment can be configured to perform various types of convolution operations, and may include a main processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520. Sixteen slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will understand that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and this disclosure embodiment is not limited in this respect.

[0057] The master processing circuit and slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections. In different application scenarios, the connection between multiple slave processing circuits can be either a hard connection arranged by hardwired lines or a logical connection configured according to, for example, microinstructions, to form a topology of various slave processing circuit arrays. The embodiments disclosed herein are not limited in this respect. The master processing circuit and slave processing circuits can cooperate with each other to achieve parallel processing.

[0058] To support computational functions, the main processing circuit and the slave processing circuit can include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit is responsible for the core computations of deep learning algorithms, such as matrix multiplication and convolution.

[0059] The processing circuit can, for example, perform intermediate operations on the corresponding data in parallel according to the operation instructions to obtain multiple intermediate results, and then transmit the multiple intermediate results back to the main processing circuit.

[0060] By configuring the computing device 500 into a master-slave structure (e.g., a master-multiple-slave structure, or a multi-master-multiple-slave structure, which is not limited in this disclosure), for forward computation instructions, the data can be split according to the computation instructions, thereby enabling multiple slave processing circuits to perform parallel computation on the computationally intensive parts to improve computation speed, save computation time, and thus reduce power consumption.

[0061] In some embodiments disclosed herein, by utilizing different data paths to transmit input feature maps and weights, multiple reuse methods for input feature maps and weights can be supported, thereby reducing data access during computation and improving processing efficiency.

[0062] Specifically, the computing device 500 may further include a first storage circuit 530 and a second storage circuit 540, for storing data transmitted via different data channels, respectively. Optionally, the first storage circuit 530 and the second storage circuit 540 may be two storage blocks formed by partitioning the same memory, or they may be two independent memories; no specific limitation is made here. The first storage circuit 530 and the second storage circuit 540 may also be collectively referred to as the input storage circuit, for storing the input feature map and the convolution kernel, respectively.

[0063] The first storage circuit 530 can be used to store multicast data, meaning that the data in the first storage circuit will be transmitted to multiple slave processing circuits via a broadcast bus, and these slave processing circuits will receive the same data. It can be understood that broadcasting and multicasting can be implemented via a broadcast bus. Multicast refers to a communication method that transmits a single data set to multiple slave processing circuits; while broadcasting is a communication method that transmits a single data set to all slave processing circuits, and is a special case of multicast. Since both multicast and broadcasting correspond to one-to-many transmission methods, this document does not specifically distinguish between the two; broadcasting and multicast can be collectively referred to as multicast, and those skilled in the art can understand their meaning from the context.

[0064] The second storage circuit 540 can be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0065] By providing a first storage circuit and a second storage circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by multiplexing multicast data among multiple slave processing circuits.

[0066] In some embodiments, the input feature map can be determined as multicast data and stored in a first storage circuit to transmit the data via broadcast to multiple scheduled slave processing circuits during computation. Correspondingly, the convolution kernel can be determined as distribution data and stored in a second storage circuit. This distribution data can be distributed to the corresponding slave processing circuits before computation.

[0067] The computing device 500 may further include an output storage circuit for storing an output feature map processed by the processing circuit. The output storage circuit may reuse the first storage circuit or the second storage circuit, or it may be a separate storage circuit; the embodiments disclosed herein are not limited in this respect.

[0068] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of this disclosure is also shown. As shown, each slave processing circuit 520 may include multiple arithmetic circuits CU 521, a first buffer circuit 522, and a second buffer circuit 523. Four arithmetic circuits CU0 to CU3 are shown in the figure. Those skilled in the art will understand that the number of arithmetic circuits may be more or less, depending on the specific hardware configuration, and the embodiments of this disclosure are not limited in this respect.

[0069] In some embodiments, the first buffer circuit 522 can be used to buffer the weights or input feature maps allocated to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to buffer the input feature maps or weights allocated to the slave processing circuit. Both buffer circuits are used to select data for computation. The data in the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540; correspondingly, the data in the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computation circuit CU 521 or broadcast to all CUs 521 within the slave processing circuit 520 during computation.

[0070] Each arithmetic circuit CU 521 is used to perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit and data rows selected from the second buffer circuit, respectively, in each arithmetic cycle.

[0071] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by reusing data as much as possible among multiple processing circuits within a single slave processing circuit.

[0072] The processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each arithmetic circuit CU 521.

[0073] Understandable, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but depending on the configuration, the storage circuits and processing circuits can also be combined into a single module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, while the second storage circuit 540 can be shared by multiple slave processing circuits 520, with each slave processing circuit allocated an independent storage area to accelerate access. This disclosure does not limit the embodiments in this respect. Furthermore, in this computing device, the main processing circuit and slave processing circuits can belong to different modules of the same processor or chip, or they can belong to different processors; this disclosure also does not limit this in this respect.

[0074] Exemplary Convolution Optimization Scheme

[0075] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is typically allocated in contiguous storage space, meaning that multidimensional data can be unfolded one-dimensionally and stored sequentially in memory. For example, input feature maps can be stored sequentially in a low-dimensional priority manner (here, C / Ci is the lowest dimension). Adjacent dimensions refer to dimensions that are immediately next to each other in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. When the storage order is consistent with the dimension order, adjacent dimensions are located consecutively in memory. Here, W and Ci are adjacent, and their data is also consecutively stored in memory.

[0076] In intelligent processors, due to computational power requirements and considerations of area and power consumption, the main hardware operation unit is the vector multiply-accumulate unit. Implementing support for various convolution algorithms in hardware design essentially involves maximizing the extraction of multiply-accumulate operations from the algorithms and implementing them in on-chip RAM (such as...) via data paths. Figure 3 It efficiently exchanges input and output data for multiplication and addition operations between NRAM, WRAM, etc. and the arithmetic unit.

[0077] Hardware storage operates in rows (cache lines). Read, write, and computation operations are most efficient when aligned to entire rows. Therefore, to fully utilize bandwidth and accommodate the memory access demands of the arithmetic unit array, data is typically vectorized and aligned. Artificial intelligence chips are usually designed with Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above, where data along the Ci dimension is contiguous. Therefore, vectorization alignment requires the Ci dimension to be aligned to a specified value, such as alignment value M, so that data access is performed in units of this alignment value M. M can also be called the maximum number of operations per hardware operation. Depending on the hardware design, M can have different values, such as 32B, 64B, 128B, etc. Typically, the input port size of the arithmetic unit array is also related to M. For example, in the case of symmetrical input data bit width, the input port size of the arithmetic unit array is usually twice M, meaning it processes input feature map data and weight data of alignment value M in a single operation. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0078] When the Ci dimension of the input feature map is small, or when the remainder after dividing Ci by M (i.e., the remainder after splitting Ci by M) is small (e.g., smaller than the size of a cache line), the Ci dimension needs to be padded to a single line of data (e.g., 64B), i.e., filled with invalid data 0. This padding causes a lot of redundant computation, leading to wasted resources and reduced computational efficiency.

[0079] A small convolution scheme suitable for cases with a small channel C has been proposed, in which the computational data is split into split units and stored in a transformed dimensional order. The amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, thereby performing computational processing on a unit-by-unit basis, which can fully utilize the hardware's computing power and avoid or reduce invalid computations.

[0080] However, in this small convolution scheme, both the input feature map and the convolution kernel need to be pre-processed by software through block division and dimensionality transformation, and the output feature map also needs to be correspondingly processed by software through block division and dimensionality transformation, which undoubtedly increases the complexity of the software. Furthermore, alignment processing is also required during these block division and dimensionality transformation processes. Moreover, these small convolution schemes only support convolution operations with a stride of 1 in both the width and height directions.

[0081] In view of this, to further optimize convolution operations and reduce software complexity, this disclosure provides a convolution scheme that segments the Ci dimension. By segmenting the input channel Ci dimension of the input feature map and splitting other dimensions according to the segmentation, the alignment requirements of the data can be reduced, and hardware performance can be fully utilized. Furthermore, the output channel Co dimension of the output feature map can also be segmented. Thus, when there are multiple consecutive convolutional layer operations, the output feature map of the current convolutional layer can be directly used as the input feature map of the next convolutional layer without additional segmentation processing, saving the software from data block and dimension transformation processing. In addition, by supporting discrete readouts, arbitrary convolution strides can be supported without changing the processing behavior of the computing circuit, without restricting the width and height inverse convolution strides to 1.

[0082] Specifically, in some embodiments, a computing device is provided, configured to perform convolution operations. The computing device includes: an input storage circuit storing an input feature map and a convolution kernel, wherein the input feature map is segmented according to a splitting granularity Pci of the input channel Ci dimension; a plurality of slave processing circuits, each configured to perform a convolution operation on a broadcast-transmitted input feature map and a corresponding convolution kernel assigned to it; and an output storage circuit storing output feature maps output by the plurality of slave processing circuits, wherein the output feature maps are segmented according to a splitting granularity Pco of the output channel Co dimension.

[0083] By simultaneously storing the input feature map in segments according to Pci and the output feature map in segments according to Pco, it is possible to directly use the output feature map of the current convolutional layer as the input feature map of the next convolutional layer in a neural network model with multiple consecutive convolutional layers, thus achieving seamless connection without the need for additional segmentation processing. The following will describe in detail the various aspects involved in convolution operations.

[0084] Example Input Feature Map Storage

[0085] In some embodiments, the output data of the layer preceding certain convolutional layers (e.g., non-first convolutional layers in U-Net) has been divided into multiple segments along the Ci dimension, with each segment having a Ci size of 16B (e.g., data type int8), 32B (e.g., data type int16), or 64B. In this case, the granularity of the input channel Ci dimension splitting, Pci, can follow the size of each segment, i.e., 16B, 32B, or 64B.

[0086] In some embodiments, when the data is not pre-segmented, the input channel splitting granularity Pci can be determined based on the size of the input channel dimension Ci of the input feature map and the amount of data M processed by the hardware in a single run. For example, in the U-net network, the Ci dimension of the data is required to be aligned to 16 numbers, so the splitting granularity Pci of the Ci dimension can also be aligned to 16 numbers. When the data type is int8 or higher, the splitting granularity Pci is aligned to 16B, that is, it can be 16B, 32B, or 64B. It can be understood that the convolution scheme of this disclosure embodiment can reduce the alignment requirement by splitting the Ci dimension according to the splitting granularity, thus making it suitable for any Ci size. In addition, it can be understood that the maximum splitting granularity Pci does not exceed the hardware's one-time processing alignment value M (or the baseline alignment value, the amount of data processed by the hardware in a single run), that is, Pci≤M. Therefore, under different value ranges of Ci, a suitable Pci can be selected, thereby reducing the alignment requirement for the Ci dimension.

[0087] In some embodiments, the input channel splitting granularity Pci can be selected as M / 2. n n = 0, 1, 2, ..., thus facilitating storage from the second lowest dimension W in 2... n The data is contributed to the lowest storage dimension Ci to meet hardware alignment requirements. Table 1 shows several exemplary schemes corresponding to the input channel splitting granularity Pci, assuming M = 64B.

[0088] Ws (W contribution) 16 8 4 2 1

[0089] Table 1

[0090] As shown in Table 1, the composition of an input feature row varies depending on the splitting granularity Pci. When Pci = 16B, the W dimension needs to contribute 4 times the data, meaning the shape of a data row is Wi*Ci = 4 × 16B. When Pci = 32B, the shape of a data row is Wi*Ci = 2 × 32B.

[0091] It can also be seen that the smaller the input channel splitting granularity, the more parts Wi contributes to the Ci direction, and the larger the alignment constraint on Wi, which needs to satisfy Wi / Ws≥1.

[0092] This is understandable, although theoretically the granularity of the split can be M / 2. n However, considering the requirements for the W dimension when the splitting granularity is too small, the instruction overhead, the actual range of Ci values, and the overhead related to subsequent discrete readings to support arbitrary convolution strides, it is advisable to choose only M / 2. n A subset of the values ​​in the table are used as alternative splitting granularities. In the example where M = 64B, alternative splitting granularities could include, for example, 64B, 32B, and 16B.

[0093] Different splitting granularities can be applied to different computational scenarios, thereby achieving varying degrees of performance optimization. Specifically, in some embodiments, the input channel splitting granularity Pci can be selected as follows:

[0094] Align the lowest storage dimension Ci of the input feature map to each of the candidate splitting granularities; and

[0095] Taking into account the alignment padding amount and the size of the corresponding splitting granularity for each candidate splitting granularity, a suitable splitting granularity is selected, such as a candidate splitting granularity with an alignment padding amount within a predetermined range and as large as possible, as the Pci.

[0096] For example, when the alignment fill amount is the same, a larger split granularity is preferred; or when the alignment fill amount is different, a split granularity with the smallest alignment fill amount is selected; or when the alignment fill amounts are not significantly different (e.g., within a predetermined range, such as no more than 16B), a larger split granularity is preferred.

[0097] Although the rules for selecting the input channel splitting granularity Pci are listed above, these rules are merely preferred embodiments for selecting the optimal input channel splitting granularity best suited to the current Ci value. The application of these rules is described below with several examples. All examples assume M = 64B, and alternative splitting granularities include 64B, 32B, and 16B.

[0098] In one example, assuming Ci = 48B, alignment to 16B requires no zero padding, while alignment to 32B and 64B both require 16B padding. In this case, the splitting granularity that does not require zero padding can be preferred as Pci, i.e., 16B.

[0099] In another example, assuming Ci = 28B, alignment to 16B and 32B both require padding with 4B zeros, and alignment to 64B requires padding with 36B zeros. In this case, a larger splitting granularity with less alignment padding can be preferred as Pci, i.e., 32B.

[0100] Figure 6The diagram illustrates a schematic storage method for an input feature map according to some embodiments of this disclosure. As shown, assuming Pci = 16B or 32B, and the number of segments Ci_seg_num is 2, the input feature map can be stored in two segments according to Ci, with each segment stored consecutively in HWC dimension order; the address interval between the two segments is Ci_seg.stride: the size of each segment Ci is 16B or 32B. The data format can be represented as [2, Hi, Wi, 16B] or [2, Hi, Wi, 32B]. For a 16B segment, the shape of a data row is Wi*Ci = 4 × 16B; for a 32B segment, the shape of a data row is Wi*Ci = 2 × 32B.

[0101] The input feature map can be stored in the first storage circuit 530 in the segmented storage manner described above, so as to be transmitted to the scheduled slave processing circuits via broadcast during the operation.

[0102] Therefore, the storage format of the input feature map in this disclosure embodiment has been described above.

[0103] Exemplary convolution kernel storage

[0104] Convolution computation involves multiplying and adding each input feature map with a convolution kernel of size Co, resulting in Co output feature maps. However, on-chip memory cannot always store convolution kernels and input feature maps of all sizes simultaneously. Therefore, hardware involves a series of operations that repeatedly load input feature data or weight data. Balancing the repeated loading of input feature data or weight data significantly impacts computational efficiency. In practical applications, to reduce frequent off-chip memory accesses, different reuse methods can be adopted based on the scale and characteristics of the data involved in the computation.

[0105] As described above regarding the principles of convolution operations, the results of operations along the Co dimension do not need to be accumulated. Therefore, operations on different Co dimensions can be performed relatively independently on different computational circuits. That is, different convolution kernels of Co can be assigned to different computational circuits, using the same input feature map for computation. In this case, the input feature map is reused among these computational circuits, with the number of reuses Rn = Ns, where Ns is the number of computational circuits.

[0106] In some embodiments disclosed herein, the Co value assigned to each slave processing circuit can be determined based on the splitting granularity Pco of the output channel Co dimension of the convolution kernel and the number of schedulable slave processing circuits Ns.

[0107] To simplify the scheduling of slave processing circuits, in some embodiments, convolutional kernels can be stored in blocks according to the Co dimension, based on the splitting granularity Pco of the output channel Co dimension, so that the scheduled slave processing circuits can load the corresponding weight blocks. In some embodiments, the splitting granularity Pco of the output channel Co dimension can be selected as a multiple of the number of schedulable slave processing circuits Ns, thereby facilitating the even distribution of Co to Ns slave processing circuits. For example, each slave processing circuit processes convolutional kernels with different Pco / Ns Co values ​​spaced at Ns intervals. In other embodiments, the splitting granularity Pco of the Co dimension can be aligned to 16 numbers, such as 16, 32, 48, 64, etc. Due to various constraints such as instruction limits, Pco typically does not exceed the maximum processing capacity of a single microinstruction, such as 64.

[0108] As mentioned earlier, in some embodiments, the convolution kernel can be designated as distribution data and stored in the second storage circuit 540 for distribution to the corresponding slave processing circuit or for reading by the slave processing circuit before computation. The second storage circuit 540 can be shared by multiple (e.g., Ns) slave processing circuits 520, with each slave processing circuit allocated an independent storage area. This ensures that the data required for computation by each slave processing circuit only needs to be read from its corresponding storage area, accelerating memory access. When the convolution kernel is partitioned and stored according to the Co dimension, the convolution kernel corresponding to the Co value allocated to a certain slave processing circuit can be stored in the corresponding storage area of ​​the second storage circuit. Since the Co dimension is the highest storage dimension of the convolution kernel, partitioning and storing along the Co dimension does not require dimension transformation or other processing.

[0109] Figure 7 A schematic diagram of a convolution kernel storage method according to an embodiment of this disclosure is shown. The diagram exemplarily illustrates eight storage regions 700-707 allocated for, for example, Ns = 8 slave processing circuits SL0-SL7. Each storage region stores the convolution kernel corresponding to the Co value to be processed by that slave processing circuit.

[0110] As mentioned earlier, the Co dimension of the convolutional kernel is split according to Pco. Assuming Pco = 48, then when Ns = 8, each slave processing circuit needs to process 6 Co values. In the example shown in the figure, consecutive Co values ​​are sequentially (i.e., in units of 1) assigned to 8 slave processing circuits (SLs), and the assignment starts again after one round. For example, the figure shows that convolutional kernels with Co = 0 to 7 are stored sequentially in 8 storage areas 700 to 707; the next convolutional kernels with Co = 8 to 15 are stored sequentially in 8 storage areas 700 to 707; and so on. Thus, after each round of computation, the Co dimension of the computation results output by each slave processing circuit is continuous. In another example, the convolutional kernels can also be divided into blocks according to Co values, with each SL processing one block, and each block including convolutional kernels with consecutive Pco / Ns Co values. For example, assuming Pco = 48, convolutional kernels with Co = 0–5 are stored in storage area 700 allocated to SL0, convolutional kernels with Co = 6–11 are stored in storage area 701 allocated to SL1, convolutional kernels with Co = 12–17 are stored in storage area 702 allocated to SL2, and so on, until convolutional kernels with Co = 42–47 are stored in storage area 707 allocated to SL7. Thus, the Co dimension processed by each SL is continuous.

[0111] As can be seen from the Co partitioning above, each slave processing circuit may need to process one or more Co-value convolutional kernels. When processing multiple Co-values, the input feature map processed by each slave processing circuit can be further reused for these multiple Co-value convolutional kernels, with a maximum reuse count rn = Pco / Ns, where rn represents the number of times the input feature map is reused within a single slave processing circuit. Factors such as hardware buffer space limitations (e.g.) can be considered. Figure 5 The size of the first and second buffer circuits in the circuit determines the number of times the input feature map can be multiplexed within a single processing circuit.

[0112] The above describes how convolutional kernels are aligned along the Co dimension and stored in blocks in the second storage circuit. Similar to the input feature map, each Co value of the convolutional kernel is also segmented and stored along the Ci dimension according to the splitting granularity Pci, which will not be repeated here.

[0113] Example of reading input feature lines

[0114] As mentioned earlier, the convolution optimization scheme provided in this disclosure embodiment supports arbitrary convolution strides in both the width and height directions, and is not limited to 1. In order to ensure that the processing behavior of the computing circuit does not change when the convolution stride is not 1, the data sent to the computing circuit can be adjusted, that is, the reading method of the input feature line can be adjusted.

[0115] Specifically, in some embodiments, when the convolution step size Sw in the W dimension of the convolution operation is 1, the composition of an input feature row is Pci * Ws (Ci * Wi), that is, it is composed of Ws data continuously read in the W dimension, and the size of each data in the Ci dimension is Pci. For example, when Pci = 16B and M = 64B, 4 columns of data in the W dimension are continuously read. Since within each segment of data stored by segment in the Ci dimension, the dimension storage order is HWC, that is, the WC dimension is continuous. When Pci = 32B, 2 columns of data in the W dimension are continuously read to form an input feature row. It can be understood that when the input operation bandwidth exceeds the capacity of one data row, such as 128B, then 2 input feature rows can be read at a time, thus avoiding the data supply becoming a bottleneck for subsequent operations.

[0116] In other embodiments, when the convolution step size Sw in the W dimension of the convolution operation is not 1, the composition of an input feature row is also Pci * Ws (Ci * Wi), but it is composed of Ws data read with an interval of (Sw - 1) data in the W dimension. The size of each data in the Ci dimension is Pci. For example, when Pci = 16B, M = 64B, and Sw = 2, data is read at an interval of one column in the W dimension, that is, 16B data is read every 16B data skipped, and 4 16B data are read to form a data row. When Pci = 32B and Sw = 2, 32B data is read every 32B data skipped, and 2 32B data are read at intervals to form a data row.

[0117] The above interval reading can be implemented through memory winding. It wires and reads data at corresponding addresses according to the reading interval (such as 16B), thus supporting simultaneous skip reading, that is, discrete data can be read out一次性 in one clock cycle. It can be understood that if the amount of interval data is smaller, more winding is required, denser wiring is needed, the winding difficulty is high, and the area will also increase, thus increasing the overhead. Usually, the instruction requires the address access on the RAM to be aligned according to 16B, so the overhead of wiring and fetching data at an interval of 16B is relatively small.

[0118] It can be understood that when Sw < Kw, this interval reading method can also have input feature row reuse, as will be described in detail in combination with the embodiments later.

[0119] A single exemplary convolution operation process within the processing circuit

[0120] When the input feature map is broadcast to the scheduled slave processing circuits, and the convolution kernels are distributed to the corresponding slave processing circuits, simultaneously, each slave processing circuit can perform convolution operations on the corresponding data of the input feature map and the convolution kernel. Then, the main processing circuit, according to a convolution optimization scheme, concatenates the operation results returned by multiple slave processing circuits to obtain the output feature map of the convolution operation between the input feature map and the convolution kernel. Specifically, multiple computational circuits (CUs) and various buffer circuits in the slave processing circuits can be utilized (see...). Figure 5 The specific convolution operation is performed by a buffer circuit. Depending on the size of the buffer circuit inside the processing circuit and the computing power limitations of the arithmetic circuit, multiple operation cycles are usually required to complete the required operation in each round of operation.

[0121] In some embodiments, the first buffer circuit can be used to buffer multiple input feature rows for which convolution operations will be performed, wherein one input feature row includes the amount of data Pci×Ws=M in the input feature map, where Ws is the contribution factor of the width dimension W, and M is the amount of data processed by the hardware in a single operation. Correspondingly, the second buffer circuit can be used to buffer the weight data for which convolution operations will be performed. Each operation circuit is used to perform a bitwise multiplication and accumulation operation on the input feature rows selected from the first buffer circuit and the expanded weight rows selected or generated from the second buffer circuit at each computation, wherein one of the expanded weight rows is formed by copying and expanding a column of data that is split or aligned to Pci along the Ci dimension of the convolution kernel into a column Ws.

[0122] For simplicity, the following description focuses on the processing of a single Co value within a single processing circuit SL. It is understood that similar processing is performed within other SLs.

[0123] As described above, in a typical convolution operation scenario, all arithmetic circuits within a single slave processing circuit compute an output feature map or a portion of the output feature map corresponding to the same output channel Co. Depending on the buffer space size of the first and second buffer circuits within the slave processing circuit SL, and the processing capability of the arithmetic circuit CU (e.g., internal registers), the slave processing circuit typically cannot compute all the output feature maps allocated to it in one operation. Therefore, the output feature blocks (or output blocks) can be divided based on the single-operation capability of the arithmetic circuit (e.g., computing Nop output points or partial sums in a single operation). Each output feature block corresponds to all N schedulable output points within a single SL. CU The single-operation capability (N) of each arithmetic circuit CU *Nop output points).

[0124] In this disclosed embodiment, as can be seen from the segmentation along the Ci dimension above, each output block includes Ws consecutive output points along the width Wo dimension. For example, when Pci = 16B, Ws = 4; when Pci = 32B, Ws = 2. When a processing circuit includes Ncu schedulable computational circuits, these Ncu computational circuits can compute Ncu output blocks in parallel. These output blocks can be arranged consecutively along the width Wo dimension and / or height Ho dimension on the output feature map.

[0125] For example, in the previous text Figure 5 Taking a single SL consisting of 4 CUs as an example, each CU can compute 1*Ws(Ho*Wo) output points or a partial sum of output points forming an output block in a single run. Therefore, a single SL can compute 4 such output blocks in a single run. These 4 output blocks can have various arrangements along the WoHo dimension of the output feature map.

[0126] Figure 8 Several exemplary methods for dividing an output block among multiple arithmetic circuits of a single slave processing circuit are shown according to embodiments of this disclosure. In this example, it is assumed that the number of arithmetic circuits Ncu = 4.

[0127] like Figure 8 As shown in (a), the output blocks can be arranged continuously along the Wo dimension. Specifically, four 1*Ws output blocks are arranged in a row along the Wo dimension, forming a 4Ws*1 (Wo*Ho) distribution, with each arithmetic circuit calculating one of the output blocks. When Ws = 4, the four arithmetic circuits simultaneously calculate an output feature map portion of size 16*1 (Wo*Ho). When Ws = 2, the four arithmetic circuits simultaneously calculate an output feature map portion of size 8*1 (Wo*Ho).

[0128] like Figure 8 As shown in (b), the output blocks can also be arranged continuously along the Ho dimension. Specifically, four 1*Ws output blocks are arranged sequentially in a row along the Ho dimension, thus forming a Ws*4 (Wo*Ho) distribution. When Ws = 4, the four operational circuits simultaneously calculate the 4*4 (Wo*Ho) size output feature map portion. When Ws = 2, the four operational circuits simultaneously calculate the 2*4 (Wo*Ho) size output feature map portion.

[0129] like Figure 8As shown in (c), the output blocks can be arranged continuously in both the Wo and Ho dimensions simultaneously. Specifically, four 1*Ws output blocks are arranged in a 2×2 pattern in the WoHo dimension, thus forming a 2Ws*2(Wo*Ho) distribution. When Ws = 4, the four arithmetic circuits simultaneously calculate the 8*2(Wo*Ho) size output feature map portion. When Ws = 2, the four arithmetic circuits simultaneously calculate the 4*2(Wo*Ho) size output feature map portion.

[0130] After assigning the splitting method for the output block, the specific calculation process of a single output point in the output block is described below.

[0131] As we know from the convolution operation principle above, the value of each convolution output point on the output feature map corresponds to the sum of the positional multiplications of the input feature map and the weights within its convolution window. That is, the value of a single output point is the sum of the positional multiplications of each part.

[0132] In some embodiments, for a single output point in the output feature map, the value of the output point can be calculated in the following order and multiple loops, wherein: the Kw and Kh dimensions of the convolution kernel are used as the inner loop to calculate the partial sum of the output point, where Kw is the width dimension of the convolution kernel and Kh is the height dimension of the convolution kernel; the convolution operation with a convolution stride Sw in the W dimension is used as the middle loop to calculate the partial sum of the output point; the number of segments Ci_seg_num that the convolution kernel splits according to Pci in the Ci dimension is used as the outer loop to calculate the partial sum of the output point; and the sums of each part are accumulated to obtain the value of the output point.

[0133] As described above, both the convolutional kernel and the input feature map are split into multiple segments along the Ci dimension according to Pci and stored segment by segment. Therefore, the number of segments Sci along the Ci dimension can be used as the outermost loop. The number of loops Nci = Ci_seg_num = ceil(Ci / Pci).

[0134] Since the convolution optimization scheme supports a non-zero convolution stride Sw, and thus provides input feature map data using an interval reading method, a W-dimensional convolution stride Sw can be used as an intermediate loop. The number of loops Nsw = Sw.

[0135] The Kw and Kh dimensions of the convolutional kernel can be used as the innermost loop. The total number of loops in the Sw rounds is Nk = Kw * Kh. The number of loops in each Sw round depends on the current Sw. The loops in the Kw dimension need to determine the number of loops and the corresponding index based on the value of Sw. For example, when Sw = 1, only one Sw round occurs, so the number of loops for the inner Kw and Kh is Nk = Kw * Kh. When Sw = 2, Kw = Kh = 3, two Sw rounds occur. In the first round, Sw_index = 0, and the convolutional kernel takes the weights of the 0th and 2nd columns (i.e., Kw_index = 0 / 2), which means 3 * 2 = 6 loops. In the second round, Sw_index = 1, and the convolutional kernel takes the weights of the 1st column (i.e., Kw_index = 1), which means 3 * 1 = 3 loops. The total number of loops is 3 * 3 = 9.

[0136] It can be seen that the total number of cycles for a single output point, Ncycle = Nci * Nk = Ci_seg_num * Kw * Kh, is that the partial sum is calculated for each data point in the W and H dimensions, and the partial sum is calculated in segments in the Ci dimension.

[0137] Furthermore, the cyclic order of the innermost Kw and Kh dimensions can be interchanged. In some embodiments, when the Ncu output blocks calculated by the Ncu arithmetic circuits are arranged consecutively in the width Wo dimension, the Kw dimension can be cycled first, followed by another Kw dimension cycle, thereby facilitating possible data reuse in the W dimension. In other embodiments, when the Ncu output blocks are consecutive in the height Ho dimension, the Kh dimension can be cycled first, followed by another Kw dimension cycle, thereby facilitating possible data reuse in the H dimension. In still other embodiments, when the Ncu output blocks are consecutive in both the width Wo and height Ho dimensions, the cyclic order of the Kw and Kh dimensions can be chosen arbitrarily. The above-described cyclic operation process will be described below with reference to specific embodiments.

[0138] It is understandable that when the width of the convolutional kernel exceeds the maximum kernel width Kmax supported by the processing circuit, it needs to be split in the Kw direction according to this maximum kernel width. In this case, in addition to the three layers of loops mentioned above, further loop processing is performed according to the split Kw, which will not be detailed here.

[0139] As mentioned earlier, in some embodiments, the input feature map data can be further reused in the H dimension, thereby further reducing memory access. Specifically, the selected input feature rows can be reused rn times each time, and each time they are multiplied and accumulated with the rn extended weight rows corresponding to the convolution kernel in the height dimension to obtain rn consecutive output blocks of the output feature map in the height dimension, where rn is determined according to the height dimension size Kh of the convolution kernel and the convolution stride Sy in the height direction of the convolution operation.

[0140] Figure 9 A schematic diagram illustrating the computation of reusing input feature map data along the H dimension according to some embodiments of this disclosure is shown.

[0141] As shown in the figure, when the same input feature data point is multiplied and accumulated along the H dimension through Kh weight points, the resulting partial sums belong to different output points. To avoid computational overflow, let's take the input feature data point <2,0> as an example. When the input feature data point <2,0> is multiplied and accumulated with the weight data point <0,0>, it's equivalent to the case of convolution window A, resulting in the first partial sum, which belongs to the output point <2,0>. When the input feature data point <2,0> is multiplied and accumulated with the weight data point <1,0>, it's equivalent to the case of convolution window B, resulting in the second partial sum, which belongs to the output point <1,0>. When the input feature data point <2,0> is multiplied and accumulated with the weight data point <2,0>, it's equivalent to the case of convolution window C, resulting in the third partial sum, which belongs to the output point <0,0>.

[0142] Therefore, the number of times the input feature map is reused in the H dimension depends on the maximum overlap of adjacent convolutional windows in the H dimension. For example, in the example above, Kh = 3, Sy = 1, the input feature data point <2,0> is simultaneously covered by the three convolutional windows corresponding to the three output points (i.e., output points <2,0>, <1,0>, and <0,0>), and thus can be reused 3 times. It can be understood that when Sy > 1, the number of reuses rn is less than Kh, rn = Kh - Sy + 1; and some data points are not covered by overlapping convolutional windows, that is, they do not need to be reused.

[0143] The above describes how to obtain the value of a single output point by repeatedly calculating the partial sum in multiple loops, and how to interleave input feature map reuse in the calculation of a single output point to calculate multiple output points / output blocks in the H dimension.

[0144] To fully utilize the parallel operation of multiple computational circuits within a single slave processing circuit, multiple computational circuits (CUs) within a single slave processing circuit compute the output feature map in parallel. Considering the dimensional storage order of the output feature map and the contribution of the input feature map (W), in some embodiments, each slave processing circuit can write the computation results of each computational circuit into a third buffer circuit in turn, following the order of width dimension Wo first, then height dimension Ho. The computing device (e.g., the main processing circuit) can then read the computation results from the third buffer circuit of each slave processing circuit according to the order of the output channel Co values ​​within a single Pco segment, so that the dimensional storage order HoWoCo is followed within each Pco segment. This achieves segmented storage of the output feature map by Pco, which can be directly used for convolution operations in the next layer.

[0145] The detailed calculation process of convolution operation in the embodiments of this disclosure is described below with reference to specific examples.

[0146] In the following example, the parameters are defined as follows:

[0147] Ci_byte: Represents the number of bytes in the Ci dimension;

[0148] Co_num: Represents the size of the Co dimension;

[0149] Ci_seg_size: indicates the size of the Ci dimension segment, which is the splitting granularity Pci mentioned above. For example, it can be 16B or 32B.

[0150] Co_seg_size: indicates the size of the Co dimension segment, which is the splitting granularity Pco mentioned above. For example, it can be 16, 32, 48, 64, etc.

[0151] Ci_seg_num: Represents the number of segments in the Ci dimension, Ci_seg_num = ceil(Ci_byte / Ci_seg_size);

[0152] Co_seg_num: Represents the number of segments in the Co dimension, Co_seg_num = ceil(Co_num / Co_seg_size).

[0153] Embodiment 1: Ci_seg_size=16B, Ci_seg_num=2, wo*ho=16*1, Co_seg_size=48, Kw =Kh=3,Sw=Sh=2

[0154] Figures 10a-10d The diagram illustrates the operation process of the convolution operation scheme according to Embodiment 1 of this disclosure. In this embodiment, Ci is split into two segments, Ci_seg_index = 0 to 1, each segment being 16 bytes. Therefore, the format of an input feature data row is 4 × 16 bytes (WiCi). The diagram shows that each row of data includes 4 columns of Wi data, so the output block calculated by one operation circuit includes 1 × 4 (CoWo) output points. Co_seg_size = 48. Assuming the number of schedulable slave processing circuits Ns = 8, each slave processing circuit processes 6 Co values ​​per round of operation. It is not unreasonable to assume that the size of the convolution kernel is KhKw = 3 × 3. In the following description, height and width dimension coordinates are used.<h,w> The data points are represented by Ci, and each data point has a size of Ci_seg_size in the Ci dimension, which is 16B in this example.

[0155] Considering the convolution stride Sw = Sh = 2, when feeding data to the first buffer circuit, the corresponding input feature map data can be read in a discrete manner, reading 16B at a time, and cached in the first buffer circuit. Assuming the input operation bandwidth is 128B, eight 16B data points can be read in one cycle. In this example, according to the convolution stride Sw = 2, the first cycle can read 8×16B of input feature map data with w-direction coordinates of 0, 2, 4, 6, 8, 10, 12, and 14, and store them in the first buffer circuit. Since Sw is not 1 in this example, there are loops caused by Sw. For example, when Sw = 2, there are two loops: an even-numbered loop in the w-direction (corresponding to Sw_index = 0) and an odd-numbered loop (Sw_index = 1).

[0156] Figure 10a The loop operation process for Ci_seg_index = 0, Sw_index = 0, Kh_index = 0, and Kw_index = 0 is shown. N is selected from the first buffer circuit according to the method corresponding to the output block partitioning. CU Each of the N input feature lines is sent to N. CU N computational circuits. During the subsequent 6 calculations, these N CU The input feature rows are respectively in N CU Each operational circuit is reused 6 times.

[0157] During these six calculations, extended weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the Ncu arithmetic circuits for computation. In this example, each extended weight row is formed by expanding 1 / 4 of the weight row by copying it three times. For example, in the example shown in the figure, assume the processing circuit is SL0, which processes six convolutional kernels with Co = 0 / 8 / 16 / 24 / 32 / 40.

[0158] During the first calculation period indicated by arrow ①, numbers are selected from the data segment where Ci_seg_index = 0. Specifically, a data row consisting of input feature points <0,0>, <0,2>, <0,4>, and <0,6> is selected and sent to the first arithmetic circuit CU0; a data row consisting of input feature points <0,8>, <0,10>, <0,12>, and <0,14> is selected and sent to arithmetic circuit CU1; a data row consisting of input feature points <0,16>, <0,18>, <0,20>, and <0,22> is selected and sent to arithmetic circuit CU2; and a data row consisting of input feature points <0,24>, <0,26>, <0,28>, and <0,30> is selected and sent to arithmetic circuit CU3 (the number selection is shown in the figure with a black dashed box). Accordingly, the extended weight row A0A0A0A0, formed by expanding the data point <0,0> (hereinafter referred to as "A0") in the convolution kernel data segment where Co=0 and Ci_seg_index=0, is selected and broadcast to the four operational circuits. The four operational circuits then perform bitwise multiplication and accumulation operations to obtain the partial sum of the 16 output points w0 to w15 at ho=0, with each operational circuit calculating the four adjacent output points. It can be understood that these calculated output points are partial sums, only accumulating the data at Ci_seg_index=0 corresponding to Kh_index=0 and Kw_index=0. During the remaining five calculation periods shown by arrows ②-⑥, the partial sums of the output points on the corresponding Co are calculated for the five convolution kernels Co=8 / 16 / 24 / 32 / 40, which will not be described in detail here. Thus, for each of the six Co values ​​processed by the processing circuit, the process is iteratively repeated to obtain the 16 Wo-dimensional output points for each of the six Co values. In the computation of each CU, the input feature rows are multiplexed 6 times, while in the computation of a single slave processing circuit, the weights are multiplexed N times. CU *M / Ci_seg_size=4*(64 / 16)=16 times.

[0159] Next, we can continue the inner loop of the Kw dimension, that is, Kw_index = 2.

[0160] Figure 10b The loop operation process for Ci_seg_index = 0, Sw_index = 0, Kh_index = 0, and Kw_index = 2 is shown. Since the convolution stride Sw = 2, the stride should also be 2 during normal sliding selection. In this embodiment, since skipping values ​​based on the convolution stride has already been performed when providing the input feature map data to the first buffer circuit, sliding selection can still be performed with a stride of 1 on the first buffer circuit. At this time, for Kw_index = 2, the same process is performed... Figure 10aThe computation process involves a similar Co loop and input feature map reuse. The difference lies in the selection of data for computation: Four corresponding input feature rows are selected by sliding one step in the Wi direction from the first buffer circuit and sent to the four computation circuits respectively; and weight data corresponding to different Co values, Kh_index=0 and Kw_index=2, are sequentially selected from the second buffer circuit. During the subsequent six calculations, these four input feature rows are reused six times in each of the four computation circuits, and each time they are processed with extended weight rows of different Co values.

[0161] For example, during the first calculation period indicated by arrow ①, the selection is performed by sliding one step from the first buffer circuit. Specifically, the data row consisting of input feature points <0,2>, <0,4>, <0,6>, and <0,8> is selected and sent to the first operation circuit CU0; the data row consisting of input feature points <0,10>, <0,12>, <0,14>, and <0,16> is selected and sent to the operation circuit CU1, and so on. Correspondingly, the extended weight row A2A2A2A2, which is expanded from the data point <0,2> (hereinafter referred to as "A2") in the convolution kernel data segment where Co=0 and Ci_seg_index=0, is selected and broadcast to the four operation circuits. Thus, the four operation circuits perform bitwise multiplication and accumulation operations respectively to obtain the partial sum of the 16 output points w0 to w15 at ho=0, and each operation circuit calculates the four adjacent output points. It's understandable that the output points of these calculations are partial sums, only accumulating the data at Ci_seg_index=0 corresponding to Kh_index=0 and Kw_index=2. It's also understandable that, since the input feature row slides in the Wi direction (Sw=2), and the weight Kw also slides 2 steps in the W direction (Kw_index=2), these partial sums still belong to the 16 output points w0~w15 in the Wo dimension where Co=0. Therefore, the partial sums obtained in this calculation are accumulated from the previous ( Figure 10a The partial sum obtained from the operation in )

[0162] During the remaining five computation periods indicated by arrows ②-⑥, the partial sums of the output points on Co are similarly calculated for the five convolution kernels Co = 8 / 16 / 24 / 32 / 40, respectively, which will not be described in detail here.

[0163] Therefore, after the loop processing of the Kw part where Sw_index=0 and Kh_index=0 is completed when Sw=2, the Kh loop can then be performed, that is, the loop switches to Kh_index=1.

[0164] Figure 10cThe loop operation is shown for Ci_seg_index = 0, Sw_index = 0, Kh_index = 1, and Kw_index = 0. Note that the first buffer circuit needs to reload the input feature map data at this point. Figure 10a Similarly, based on the convolution stride Sw=2, the first step can read 8×16B input feature map data with w-direction coordinates of 0, 2, 4, 6, 8, 10, 12, 14 respectively. The difference is that the data in the row with hi=1 is read at this time.

[0165] from Figure 10c As can be seen, similarly, through 6 calculation cycles, the partial sums of the 16 output points w0 to w15 at ho=0 for each of Co = 0 / 8 / 16 / 24 / 32 / 40 can be calculated, with each operational circuit calculating 4 adjacent output points. It can be understood that these calculated output points are partial sums, only accumulating the data at Ci_seg_index=0 corresponding to Kh_index=1 and Kw_index=0. It can also be understood that since hi=1 and the weight Kh_index=1 for the input feature row, these partial sums still belong to the 16 output points w0 to w15 on the Wo dimension of the corresponding Co. Therefore, the partial sums obtained in this calculation are accumulated in the previous ( Figure 10b The partial sum obtained from the operation in )

[0166] Similarly, the aforementioned loop operation can be performed for Ci_seg_index=0, Sw_index=0, Kh_index=1, Kw_index=2, thereby completing the loop for the part where Kh_index=1.

[0167] After the Kh loop finishes processing (for example, continuing to perform calculations on the corresponding data when Ci_seg_index=0, Sw_index=0, and Kh_index=2), the Sw loop can then be executed, i.e., switching to Sw_index=1. This loop corresponds to the odd-numbered columns of data in the w direction of the input feature map.

[0168] Figure 10d The loop operation process is shown for Ci_seg_index = 0, Sw_index = 1, Kh_index = 0, and Kw_index = 1. Note that the input feature map data loaded by the first buffer circuit at this time consists of w-direction coordinates of 1, 3, 5, 7, 9, 11, 13, 15, ...

[0169] During the first calculation period indicated by arrow ①, numbers are selected from the data segment where Ci_seg_index = 0. Specifically, the data row consisting of input feature points <0,1>, <0,3>, <0,5>, and <0,7> is selected and sent to the first operational circuit CU0; the data row consisting of input feature points <0,9>, <0,11>, <0,13>, and <0,15> is selected and sent to the operational circuit CU1, and so on. Correspondingly, the extended weight row A1A1A1A1, which is expanded from the data point <0,1> (hereinafter referred to as "A1") in the convolution kernel data segment where Co = 0 and Ci_seg_index = 0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively to obtain the partial sum of the 16 output points w0 to w15 at ho = 0, and each operational circuit calculates the four adjacent output points. It is understandable that the output points of these calculations are partial sums, only accumulating the data at Ci_seg_index=0 corresponding to Kh_index=0 and Kw_index=1.

[0170] During the remaining five computation periods indicated by arrows ②-⑥, the partial sums of the output points on Co are calculated for the five convolution kernels Co = 8 / 16 / 24 / 32 / 40, respectively. These will not be described in detail here.

[0171] Therefore, based on the split of Sw, all loops in the Kw direction have been processed.

[0172] Next, the aforementioned loop operation process can be performed for Kh_index=1 and 2 when Ci_seg_index=0 and Sw_index=1 respectively, thereby completing the loop of Kh part and also completing the loop of Sw.

[0173] Next, a Ci_seg loop can be performed, that is, switching to Ci_seg_index=1, and repeatedly executing the above number selection and loop calculation process from the data segment with Ci_seg_index=1. After all these loop processing is completed, each operation circuit can accumulate to obtain the final convolution result of 4 output points for each of the 6 Co. The 4 operation circuits in one slave processing circuit summarize to obtain 16 output points in the wo direction for each of the 6 Co. The 8 slave processing circuits obtain a total of 16*1 (Wo*Ho) output points for each of the 48 Co.

[0174] Figure 11 A schematic diagram of the writing and output logic of the calculation results according to Embodiment 1 of this disclosure is shown.

[0175] As shown in the figure, multiple arithmetic circuits (CUs) within a single slave processing circuit (SL) can sequentially write the computation results into a result buffer circuit (e.g., according to the computation order). Figure 5In the third buffer circuit. Specifically, the output points of the same Co calculated by each CU can be written first in the order of Wo (writing loop ①). Then, the output points of different Co calculated by each CU can be written in the order of Co (writing loop ②). For example, for SL0, first write w0~w15 when Co=0, then write w0~w15 when Co=8, then w0~w15 when Co=16, and so on. Similar results are written in other SLs, except that the Co values ​​are processed differently.

[0176] As can be seen from the above writing order, the operation results in the result buffer circuit are stored in CW (W in the lowest storage dimension) order. However, the final output result is expected to be stored in segments according to Co, and within each segment, it is stored in HWC order (C in the lowest storage dimension). Therefore, in the data path for reading the result from the result buffer circuit, the data can be converted from CW order to WC order.

[0177] Specifically, as shown in the figure, the first output point w0 of the W dimension can be read from the result buffer circuit of each processing circuit in the order of Co (output loop ①), and then the output points on each Wo can be read in the order of Wo (output loop ②). Figure 11 The right-hand view shows the readout results. Note that when reading in Co order, the results are read in turn on the 8 SL result buffer circuits so that the Co dimensions are continuous, for example from 0 to 47.

[0178] Therefore, the data output segmented by Co can be directly used for the operation of the next convolutional layer without the need for additional pendulum processing.

[0179] Embodiment 2: Ci_seg_size=16B, Ci_seg_num=2, wo*ho=4*4, Co_seg_size=48, Kw =Kh=3,Sw=Sh=1

[0180] Figure 12 A schematic diagram of the operation process of the convolution operation scheme according to Embodiment 2 of this disclosure is shown. Embodiment 2 has most of the same parameters as Embodiment 1, the difference being that the output data processed in a single processing circuit is split into wo*ho = 4*4, and the convolution stride is 1.

[0181] Since the convolution stride Sw = Sh = 1, the input feature maps in the w direction can be read directly and cached in the first buffer circuit when feeding data to it. Furthermore, since the output data is split into 4*4 (wo*ho) values, data also needs to be read in the h direction. In this example, the first iteration can read a total of 8×16B of input feature map data with h-direction coordinates of 0–1 and w-direction coordinates of 0–3; the second iteration can read a total of 8×16B of input feature map data with h-direction coordinates of 2–3 and w-direction coordinates of 0–3. In this example, Sw is 1, therefore there is no loop caused by Sw, meaning Sw_index has only a value of 0.

[0182] Figure 12 The loop operation process for Ci_seg_index = 0, Sw_index = 0, Kh_index = 0, and Kw_index = 0 is shown. N is selected from the first buffer circuit according to the method corresponding to the output block partitioning. CU Each of the N input feature lines is sent to N. CU N computational circuits. During the subsequent 6 calculations, these N CU The input feature rows are respectively in N CU Each of the six operational circuits is multiplexed six times. During these six calculations, extended weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the Ncu operational circuits for calculation.

[0183] During the first calculation period indicated by arrow ①, numbers are selected from the data segment where Ci_seg_index = 0. Specifically, a data row consisting of input feature points <0,0>, <0,1>, <0,2>, and <0,3> is selected and sent to the first operational circuit CU0; a data row consisting of input feature points <1,0>, <1,1>, <1,2>, and <1,3> is selected and sent to the operational circuit CU1, and so on (the number selection is shown in the figure with a black dashed box). Correspondingly, the extended weight row A0A0A0A0, which is expanded from the data point <0,0> (“A0”) in the convolution kernel data segment where Co = 0 and Ci_seg_index = 0, is selected and broadcast to the four operational circuits. Thus, the four operational circuits perform bitwise multiplication and accumulation operations respectively to obtain the partial sum of the 16 output points w0 to w3 at ho = 0 to 3. Each operational circuit calculates the four adjacent output points at different ho. It is understandable that the output points of these calculations are partial sums, only accumulating the data at Ci_seg_index=0 corresponding to Kh_index=0 and Kw_index=0. During the remaining five calculation periods shown by arrows ②-⑥, the partial sums of the output points on Co are calculated for the five convolutional kernels Co=8 / 16 / 24 / 32 / 40 respectively, which will not be described in detail here. Thus, for each of the six Co values ​​processed by the processing circuit, the process is iteratively repeated to obtain 4*4 (wo*ho) output points for each of the six Co values.

[0184] Since the data in the h-direction has already been loaded, we can consider looping Kh times in the h-direction according to the size of Kh to calculate the partial sums, thereby achieving the reuse of the input feature map. In this case, looping along the Kh dimension, that is, sequentially sliding the data in the h-direction according to Kh_index = 1, 2, etc., to perform the calculation of the partial sums. For example, for Kh_index = 1, the data rows with wi = 0 to 3 at hi = 1, 2, 3, 4 are selected and sent to four processing circuits to calculate the partial sums of the four output points w0 to w3 at different ho.

[0185] After the iterative processing in the Kh dimension, the iterative processing in the Kw dimension can continue. That is, the partial sums are calculated sequentially by taking the corresponding numbers in the w direction according to Kw_index = 1 and 2. For example, for Kw_index = 1, the selected input feature map data is the data rows of wi = 1 to 4 on hi = 0, 1, 2, 3, which are sent to 4 arithmetic circuits to calculate the partial sums of the 4 output points w0 to w3 on ho = 0 to 3 respectively.

[0186] After the loop processing in the Kw dimension is completed, since the convolution stride is 1, there is no loop caused by the convolution stride.

[0187] Next, a Ci_seg loop can be performed, that is, switching to Ci_seg_index = 1, and repeatedly executing the above number selection and loop calculation process from the data segment with Ci_seg_index = 1. After all these loop processing are completed, each operation circuit can accumulate to obtain the final convolution result of 4 output points on each of the 6 Co. The 4 operation circuits in one slave processing circuit summarize to obtain 4*4 (wo*ho) output points on each of the 6 Co. The 8 slave processing circuits obtain a total of 48 output points on each of the 4*4 (wo*ho) Co.

[0188] Embodiment 3: Ci_seg_size=32B, Ci_seg_num=2, wo*ho=8*1, Co_seg_size=48, Kw =Kh=3,Sw=Sh=2

[0189] Figure 13 The diagram illustrates the operation process of the convolution operation scheme according to Embodiment 3 of this disclosure. Embodiment 3 shares most of the same parameters as Embodiment 1, except that the segment size of the Ci dimension is 32 bytes, and correspondingly, the splitting method for the output data processed by a single processing circuit is wo*ho = 8*1.

[0190] In this embodiment, the Ci segment size is 32B, so the format of an input feature data row is 2×32B (WiCi). The figure shows that each row of data includes 2 columns of Wi data, so the output block of a computation circuit includes 1×2 (CoWo) output points.

[0191] Considering the convolution stride Sw = Sh = 2, when feeding data to the first buffer circuit, the corresponding input feature map data can be read in a discrete manner, reading 32B at a time per jump, and cached in the first buffer circuit. Assuming the input operation bandwidth is 128B, four 32B data points can be read in one cycle. In this example, according to the convolution stride Sw = 2, the first cycle can read 4×32B input feature map data with coordinates 0, 2, 4, and 6 in the w direction, and store them in the first buffer circuit. Since Sw is not 1 in this example, there are loops caused by Sw. For example, when Sw = 2, there are two loops: an even-numbered column loop in the w direction (corresponding to Sw_index = 0) and an odd-numbered column loop (Sw_index = 1).

[0192] Figure 13 The loop operation process for Ci_seg_index = 0, Sw_index = 0, Kh_index = 0, and Kw_index = 0 is shown. N is selected from the first buffer circuit according to the method corresponding to the output block partitioning. CU Each of the N input feature lines is sent to N. CU N computational circuits. During the subsequent 6 calculations, these N CU The input feature rows are respectively in N CUEach operational circuit is reused 6 times.

[0193] During these six calculations, extended weight rows corresponding to different Co values ​​are selected from the second buffer circuit and broadcast to the Ncu arithmetic circuits for computation. In this example, each extended weight row is formed by expanding half a weight row by copying it once. For example, in the example shown in the figure, assume the processing circuit is SL0, which processes six convolutional kernels with Co = 0 / 8 / 16 / 24 / 32 / 40.

[0194] During the first calculation period indicated by arrow ①, numbers are selected from the data segment where Ci_seg_index = 0. Specifically, the data row consisting of input feature points <0,0> and <0,2> is selected and sent to the first operational circuit CU0; the data row consisting of input feature points <0,4> and <0,6> is selected and sent to operational circuit CU1; the data row consisting of input feature points <0,8> and <0,10> is selected and sent to operational circuit CU2; and the data row consisting of input feature points <0,12> and <0,14> is selected and sent to operational circuit CU3 (the number selection is shown in the figure with a black dashed box). Correspondingly, the extended weight row A0A0, which is formed by expanding the data point <0,0> (hereinafter referred to as "A0") from the convolution kernel data segment where Co = 0 and Ci_seg_index = 0, is selected and broadcast to the four operational circuits. Therefore, the four operational circuits perform bitwise multiplication and accumulation operations respectively, obtaining the partial sums of the eight output points w0 to w7 at ho = 0. Each operational circuit calculates two adjacent output points. It can be understood that these calculated output points are partial sums, only accumulating the data at Ci_seg_index = 0 corresponding to Kh_index = 0 and Kw_index = 0. During the remaining five calculation periods shown by arrows ②-⑥, the partial sums of the output points corresponding to the five convolution kernels Co = 8 / 16 / 24 / 32 / 40 are calculated respectively, which will not be described in detail here. Thus, for the six Co values ​​processed by a single slave processing circuit, the processing is cyclically repeated to obtain the eight Wo-dimensional output points for each of the six Co values. In the calculation of each CU, the input feature rows are reused six times, while in the operation of a single slave processing circuit, the weights are reused N times. CU *M / Ci_seg_size=4*(64 / 32)=8 times.

[0195] Next, we can continue the inner loop of the Kw dimension, that is, Kw_index = 2.

[0196] Therefore, after the loop processing of the Kw part where Sw_index=0 and Kh_index=0 is completed when Sw=2, the Kh loop can then be performed, that is, the loop switches to Kh_index=1.

[0197] After the Kh loop finishes processing (for example, continuing to perform calculations on the data corresponding to Ci_seg_index=0, Sw_index=0, and Kh_index=2), the Sw loop can then be executed, i.e., switching to Sw_index=1. This loop corresponds to the odd-numbered columns of data in the w direction of the input feature map. The specific loop process is similar to the previous one and will not be described in detail again.

[0198] After completing the Sw loop, the Ci_seg loop can then begin, i.e., switching to Ci_seg_index = 1, and repeating the above number selection and loop calculation process from the data segment with Ci_seg_index = 1. After all these loops are completed, each computational circuit can accumulate to obtain the final convolution result of 2 output points for each of the 6 Co axes. The 4 computational circuits within the slave processing circuit summarize to obtain 8 output points in the wo direction for each of the 6 Co axes. The 8 slave processing circuits obtain a total of 8*1 (Wo*Ho) output points for each of the 48 Co axes.

[0199] The above description and explanation of the convolution optimization scheme provided in this disclosure, combined with the specific convolution operation process of several embodiments, have provided an exemplary description and explanation. It is understood that, depending on the different values ​​of the various parameters, there can be many more combinations to obtain different embodiments. Furthermore, based on the teachings of this disclosure, those skilled in the art can conceive of other convolution optimization schemes according to specific hardware circuit configurations (such as the number of processing circuits, the number of operational circuits within the processing circuits, and the hardware's single-processing capability), all of which fall within the scope of this disclosure and will not be listed here.

[0200] This disclosure also provides a chip that may include the data processing apparatus of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a circuit board that may include the aforementioned chip.

[0201] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0202] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0203] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0204] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0205] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0206] The foregoing can be better understood in accordance with the following terms:

[0207] Clause 1. A computing device configured to perform a convolution operation, the computing device comprising:

[0208] An input storage circuit stores input feature maps and convolutional kernels, wherein the input feature maps are segmented and stored according to the splitting granularity Pci of the input channel Ci dimension;

[0209] Multiple slave processing circuits are configured to perform convolution operations on the broadcast-transmitted input feature map and corresponding convolution kernels assigned to each slave processing circuit, respectively; and

[0210] An output storage circuit is used to store the output feature maps of the plurality of processing circuits, wherein the output feature maps are stored in segments according to the splitting granularity Pco of the output channel Co dimension.

[0211] Clause 2. The computing device according to Clause 1, wherein each slave processing circuit includes a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein:

[0212] The first buffer circuit is used to buffer multiple input feature lines that will be used to perform convolution operations. One of the input feature lines includes the amount of data Pci×Ws=M in the input feature map, where Ws is the contribution multiple of the width W dimension and M is the amount of data processed by the hardware in a single operation.

[0213] The second buffer circuit is used to cache the weight data for the convolution operation to be performed; and

[0214] Each of the aforementioned computational circuits is configured to perform a bitwise multiplication and accumulation operation on each computation for an input feature row selected from the first buffer circuit and an extended weight row selected or generated from the second buffer circuit, respectively, wherein one of the extended weight rows is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolutional kernel into a Ws column.

[0215] Clause 3. The computing device according to Clause 2, wherein:

[0216] When the convolution operation has a stride Sw of 1 in the W dimension, one input feature row consists of Ws data points read consecutively in the W dimension, with each data point having a Ci dimension of Pci; and / or

[0217] When the convolution operation has a stride Sw of not 1 in the W dimension, an input feature row consists of Ws data points read at intervals of (Sw-1) data points in the W dimension, and the Ci dimension of each data point is Pci.

[0218] Clause 4. The computing device according to any one of Clauses 2-3, wherein for a single output channel Co in the output feature map, each of the processing circuits is further configured to:

[0219] Within each computation cycle, Ncu output blocks on the output feature map are computed in parallel using Ncu schedulable computation circuits. Each output block includes Ws output points that are continuous in the width Wo dimension, and the Ncu output blocks are continuous in the width Wo dimension and / or the height Ho dimension.

[0220] Clause 5. The computing device according to Clause 4, wherein for a single output point on a single output channel Co in an output feature map, the arithmetic circuit calculates the value of the output point in a multi-level cyclical manner in the following order:

[0221] The Kw and Kh dimensions of the convolution kernel are used as the inner loop to calculate the partial sum of the output points. Kw is the width dimension of the convolution kernel, and Kh is the height dimension of the convolution kernel.

[0222] The convolution operation uses a convolution stride Sw in the W dimension as the intermediate loop to calculate the partial sum of the output points;

[0223] The number of segments Ci_seg_num that the convolution kernel splits along the Ci dimension according to Pci is used as the partial sum of the output points calculated by the outer loop; and

[0224] The sum of each part is accumulated to obtain the value of the output point.

[0225] Clause 6. The computing device according to Clause 5, wherein in the inner loop,

[0226] When the Ncu output blocks are consecutive in the width Wo dimension, prioritize looping along the Kw dimension, then loop along the Kw dimension again; or

[0227] When the Ncu output blocks are consecutive in the height Ho dimension, prioritize cycling along the Kh dimension, then cycle along the Kw dimension; or

[0228] When the Ncu output blocks are consecutive in both the width Wo and height Ho dimensions, the cyclic order of the Kw and Kh dimensions can be chosen arbitrarily.

[0229] Clause 7. The computing device according to any one of Clauses 5-6, wherein in the inner loop, the Kw dimension loop determines the number of iterations and the corresponding index based on the value of Sw.

[0230] Clause 8. The computing device according to any one of Clauses 5-7, wherein the convolution kernels are stored in blocks according to the output channel Co value for allocation to different slave processing circuits, wherein each slave processing circuit is assigned to process convolution kernels with different Pco / Ns Co values ​​spaced at Ns intervals, where Ns is the number of schedulable slave processing circuits.

[0231] Clause 9. The computing device according to Clause 8, wherein in the inner loop, each of the slave processing circuits is further configured to:

[0232] Each input feature row is multiplexed Pco / Ns times to compute Ws output points on the same width Wo dimension of the Pco / Ns output channels Co in the output feature map.

[0233] Clause 10. A computing device according to any one of Clauses 2-9, wherein each of the slave processing circuits further includes a third buffer circuit, and the slave processing circuit is further configured to:

[0234] Following the order of width dimension Wo first, then height dimension Ho, the calculation results of each calculation circuit are sequentially written into the third buffer circuit; and

[0235] The computing device is further configured to: read the computation results from the third buffer circuit of each processing circuit according to the order of the output channel Co values ​​within a single Pco segment, so that the HoWoCo dimensional storage order is followed within each Pco segment.

[0236] Clause 11. A chip comprising a computing device according to any one of Clauses 1-10.

[0237] Clause 12, a board including the chip described in Clause 11.

[0238] Clause 13. A method for performing convolution operations using any of the computing devices described in Clauses 1-10.

[0239] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A computing device configured to perform a convolution operation, the computing device comprising: The input storage circuit stores the input feature map and the convolution kernel. The input feature map is stored in segments according to the splitting granularity Pci of the input channel Ci dimension. Within each segment, the data is stored continuously in the order of HWC dimensions, where H is the height dimension, W is the width dimension, and C is the channel dimension. Multiple slave processing circuits are configured to perform convolution operations on the input feature map transmitted via broadcast and the corresponding convolution kernels assigned to the slave processing circuits, respectively. as well as An output storage circuit is used to store the output feature maps output by the plurality of processing circuits, wherein the output feature maps are stored in segments according to the splitting granularity Pco of the output channel Co dimension, and each segment is stored consecutively according to the dimension order of HWC.

2. The computing device according to claim 1, wherein each slave processing circuit includes a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein: The first buffer circuit is used to buffer multiple input feature lines that will be used to perform convolution operations. One of the input feature lines includes the amount of data Pci×Ws=M in the input feature map, where Ws is the contribution multiple of the width W dimension and M is the amount of data processed by the hardware in a single operation. The second buffer circuit is used to cache the weight data for the convolution operation to be performed; as well as Each of the aforementioned computational circuits is configured to perform a bitwise multiplication and accumulation operation on each computation for an input feature row selected from the first buffer circuit and an extended weight row selected or generated from the second buffer circuit, respectively, wherein one of the extended weight rows is formed by copying and expanding a column of data blocks split or aligned to Pci in the Ci dimension of the convolutional kernel into a Ws column.

3. The computing device according to claim 2, wherein: When the convolution operation has a stride Sw of 1 in the W dimension, one input feature row consists of Ws data points read consecutively in the W dimension, with each data point having a Ci dimension of Pci; and / or When the convolution operation has a stride Sw of not 1 in the W dimension, an input feature row consists of Ws data points read at intervals of (Sw-1) data points in the W dimension, and the Ci dimension of each data point is Pci.

4. The computing device of claim 2, wherein for a single output channel Co in the output feature map, each of the processing circuits is further configured to: Within each computation cycle, Ncu output blocks on the output feature map are computed in parallel using Ncu schedulable computation circuits. Each output block includes Ws output points that are continuous in the width Wo dimension, and the Ncu output blocks are continuous in the width Wo dimension and / or the height Ho dimension.

5. The computing device according to claim 4, wherein for a single output point on a single output channel Co in the output feature map, the arithmetic circuit calculates the value of the output point in the following order and in multiple layers of loops, wherein: The Kw and Kh dimensions of the convolution kernel are used as the inner loop to calculate the partial sum of the output points. Kw is the width dimension of the convolution kernel, and Kh is the height dimension of the convolution kernel. The convolution operation uses a convolution stride Sw in the W dimension as the intermediate loop to calculate the partial sum of the output points; The number of segments Ci_seg_num split along the Ci dimension of the convolution kernel according to Pci is used as the partial sum of the output points calculated by the outer loop; and The sum of each part is accumulated to obtain the value of the output point.

6. The computing device according to claim 5, wherein in the inner loop, When the Ncu output blocks are consecutive in the width Wo dimension, prioritize looping along the Kw dimension, then loop along the Kw dimension again; or When the Ncu output blocks are consecutive in the height Ho dimension, prioritize cycling along the Kh dimension, then cycle along the Kw dimension; or When the Ncu output blocks are consecutive in both the width Wo and height Ho dimensions, the cyclic order of the Kw and Kh dimensions can be chosen arbitrarily.

7. The computing device of claim 5, wherein in the inner loop, the Kw-dimensional loop determines the number of iterations and the corresponding index based on the value of Sw.

8. The computing device of claim 5, wherein the convolution kernels are stored in blocks according to the output channel Co value for allocation to different slave processing circuits, wherein each slave processing circuit is assigned to process convolution kernels with different Pco / Ns Co values ​​spaced at Ns intervals, wherein Ns is the number of schedulable slave processing circuits.

9. The computing device of claim 8, wherein in the inner loop, each of the slave processing circuits is further configured to: Each input feature row is multiplexed Pco / Ns times to compute Ws output points on the same width Wo dimension of the Pco / Ns output channels Co in the output feature map.

10. The computing device according to any one of claims 2-9, wherein each of the slave processing circuits further includes a third buffer circuit, and the slave processing circuit is further configured to: Following the order of width dimension Wo first, then height dimension Ho, the calculation results of each calculation circuit are sequentially written into the third buffer circuit; and The computing device is further configured to: read the computation results from the third buffer circuit of each processing circuit according to the order of the output channel Co values ​​within a single Pco segment, so that the HoWoCo dimensional storage order is followed within each Pco segment.

11. A chip comprising a computing device according to any one of claims 1-10.

12. A circuit board comprising the chip according to claim 11.

13. A method for performing convolution operations using a computing device according to any one of claims 1-10.

Images