Computing device, method of performing convolution operation using computing device, and related products

Abstract

The present disclosure discloses a computing device, a method for performing convolution operation by using the computing device, and related products. The computing device can be included in a combined processing device, which can further include an interface device and other processing devices. The computing device interacts with the other processing devices to jointly complete a user-specified computing operation. The combined processing device can further include a storage device connected to the computing device and the other processing devices respectively, for storing data of the computing device and the other processing devices. The scheme of the present disclosure optimizes the convolution operation and improves the operation processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a computing device configured to perform convolution operations, a method for performing convolution operations using the computing device, a chip, and a board. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Neural networks are among the most critical technologies in artificial intelligence and deep learning, with Convolutional Neural Networks (CNNs) being the most important type. The most crucial computation in CNNs is the convolution operation within the convolutional layers (Conv layers). The function of convolutional layers is to extract features from input data. Through multiple convolutions, complex features can be extracted to ensure the network has sufficient expressive and generalization capabilities. Neural network models contain numerous convolution operations of various types, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. When neural network models are applied to different fields, such as speech recognition, machine translation, and image processing, the corresponding input feature maps and the dimensions of the weights may vary. To fully utilize the hardware advantages of deep learning processors, optimization is needed for convolution operations of different scales and types to improve the computational performance of the neural network model. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes a computing device in several aspects that, by segmenting the input feature map and weights into blocks, enables data of various dimensional dimensions to be adapted to the hardware of convolution operations, thereby improving the computational efficiency of convolution operations. The convolution operations in embodiments of this disclosure can be operations within various neural network models applicable to various fields, such as image processing, speech processing, text processing, etc., and these processes may include, but are not limited to, recognition and classification.

[0005] In a first aspect, embodiments of this disclosure provide a computing device configured to perform convolution operations. The computing device includes a block circuit configured to: split an input feature map and a convolution kernel into multiple corresponding splitting units according to a convolution splitting scheme, wherein each splitting unit includes data in a minimum storage dimension and at least one other storage dimension, and the data volume of a splitting unit does not exceed the maximum single-operation capacity of the hardware; and convert the dimensional storage order of the input feature map and the convolution kernel such that the data within a splitting unit is stored continuously as a data row, wherein the split and converted input feature map and / or convolution kernel is provided to a main processing circuit or a slave processing circuit; the main processing circuit is configured to distribute the data it obtains to multiple slave processing circuits for performing convolution operations; and according to the convolution splitting scheme, concatenate the operation results returned by the multiple slave processing circuits to obtain an output feature map of the convolution operation of the input feature map and the convolution kernel; and the multiple slave processing circuits are configured to perform convolution operations based on the data they obtain and return the operation results to the main processing circuit.

[0006] In a second aspect, embodiments of this disclosure provide a chip that includes a computing device according to any of the embodiments of the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip of any of the embodiments of the second aspect above.

[0008] In a fourth aspect, embodiments of this disclosure provide a method for performing convolution operations using a computing device according to any of the embodiments of the first aspect.

[0009] By utilizing the computing device, chip, board, and method for performing convolution operations as described above, the solutions in this disclosure generally apply a convolution splitting scheme, enabling the input feature map to adapt to the processing capabilities of the hardware computing device. This fully utilizes the parallel processing capabilities of multiple processing circuits, effectively improving the computational efficiency of convolution operations. Furthermore, in some embodiments, the input feature map and weights can be transmitted through different data paths, supporting multiple reuse methods for the input feature map and weights, further optimizing convolution operations and reducing data access overhead. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0012] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0013] Figure 3 A schematic diagram illustrating the internal structure of a processor core in a single-core or multi-core computing device according to embodiments of the present disclosure;

[0014] Figures 4a-4c This document illustrates several exemplary examples of convolution operation principles that can be applied to embodiments of this disclosure;

[0015] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0016] Figure 6 An exemplary data storage order according to an embodiment of this disclosure is shown;

[0017] Figures 7a-7d Several exemplary grouping patterns according to embodiments of this disclosure are shown;

[0018] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of this disclosure is shown;

[0019] Figures 9a-9d A schematic diagram of data storage in a second storage circuit according to an embodiment of this disclosure is shown;

[0020] Figures 10a-10b A schematic diagram showing the output point division of an arithmetic circuit according to an embodiment of this disclosure is shown;

[0021] Figure 11 A schematic diagram illustrating the splitting and storage of the Forward16 scheme according to an embodiment of this disclosure is shown;

[0022] Figure 12 This diagram illustrates a single operation in the Forward16 scheme according to an embodiment of this disclosure;

[0023] Figure 13 This diagram illustrates sliding convolution in the Forward16 scheme according to an embodiment of this disclosure;

[0024] Figure 14 This diagram illustrates the accumulation of sliding convolution results in the Forward16 scheme according to an embodiment of this disclosure; and

[0025] Figure 15 A schematic diagram illustrating the output data format of the Forward16 splitting scheme according to an embodiment of this disclosure is shown. Detailed Implementation

[0026] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0028] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0030] Exemplary hardware environment

[0031] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0032] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0033] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0034] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0035] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0036] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0037] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0038] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0039] Figure 3 The diagram shows the internal structure of the processing core when the computing device 201 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, a processing module 32, and a storage module 33.

[0040] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0041] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0042] The storage module 33 is used to store or move relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. The NRAM 331 stores the input neurons, output neurons, and intermediate results after computation; the WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; the DMA 333 is connected to the DRAM 204 via bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204.

[0043] Exemplary convolution operation types

[0044] Based on the aforementioned hardware environment, in one aspect, this disclosure provides a computing device configured to perform convolution operations, thereby optimizing convolution operations, for example, in neural network models. Convolutional layers in a neural network model can perform convolution operations by applying convolution kernels (also called filters, weights, etc.) to an input feature map (also called input data, neurons, or input neurons) to extract features. A convolutional layer may contain multiple convolution kernels, each element of which corresponds to a weight coefficient and a bias.

[0045] Neural network models may contain various convolutional layers, such as convolutional layers that perform forward, regular 3D convolutional operations, and deconvolutional layers that perform depthwise convolutional operations. In reverse training, it may be necessary to perform inverse depthwise convolutional operations or cross-product convolutional operations. The embodiments disclosed herein can be optimized for these different types of convolutional operations.

[0046] In conventional 3D convolution operations, assuming the input feature map tensor shape in the convolutional layer is represented by X[N Hi Wi Ci], the tensor shape of the convolutional kernel is represented by K[Co Kh Kw Ci], and the output is Y[N HoWo Co], then the simplified mathematical formula for convolution operations can be expressed as follows:

[0047] Y in，jc，jh，jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0048] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, and sh and sw are the strides in the length and width directions. The formula ignores the bias, padding, and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N and C dimensions. The forward computation of the neural network model is independent in the N dimension, but fully connected in the C dimension. When the convolution kernel works, it scans the input features with a certain stride, performs matrix element-wise multiplication and sums the bias within the convolution window. In conventional 3D convolution operations, the results of the alignment multiplications in the H, W, and Ci directions are accumulated, hence the name 3D convolution. However, this 3D convolution has a constraint: the Ci dimension of the convolution kernel is equal to the Ci dimension of the input feature map, so the convolution kernel does not slide in the Ci direction, making it a pseudo-3D convolution. To distinguish it from other convolution operations in this paper, the above convolution operation is referred to as 3D convolution operation.

[0049] Figure 4a An example of a conventional 3D convolution operation principle that can be applied to embodiments of this disclosure is shown.

[0050] The figure exemplifies a four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N Hi×Wi×Ci 3D rectangles 410a. The figure also exemplifies a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co Kh×Kw×Ci 3D convolution kernels 420a. The convolution result of the input data X and the convolution kernel K yields the output data Y, which is a four-dimensional data of size [N Ho Wo Co], which can be represented as N Ho×Wo×Co 3D rectangles 430a.

[0051] The figure also shows a specific example of a convolution operation, where the input data is a 6×6×3 input feature map 440a, omitting the N dimension; the convolution kernel is a 3×3×3 stereo convolution kernel 450a, targeting a single convolution kernel Co; and the output data is a 4×4 output feature map 460a. The specific operation process is as follows:

[0052] The convolution kernel 450a scans across the input feature map 440a with a certain stride, performs element-wise matrix multiplication on the input features within the convolution window 470a, and then sums the bias values. That is, the value at each position in the output feature map 460a is obtained by performing a two-dimensional convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) in the output feature map 460a (i.e., the convolution output point) is obtained by performing a two-dimensional convolution operation between the convolution window 470a (framed by the black cube in the input feature map) and the 3D convolution kernel 450a, resulting in three values, which are then summed to obtain the final value.

[0053] To obtain outputs at other locations, the position of the convolution kernel 450a can be moved on the input feature map 440a, which means moving the convolution window of the output point. In the example in the figure, the convolution stride (Sx, Sy) is (1, 1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0, 1) or (1, 0) on the output feature map 460a can be obtained respectively.

[0054] As described above, a convolutional layer in a neural network contains N sets of input feature maps, each containing Hi × Wi × Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output feature map contains Ho × Wo × Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation also involves the convolution stride (Sx, Sy), the size of which affects the size of the output feature map.

[0055] Figure 4b An exemplary example of the principle of depthwise convolution operations that can be applied to embodiments of this disclosure is shown.

[0056] The difference between depthwise convolution and regular 3D convolution is that the latter does not accumulate along the depth direction, which refers to the input channels Ci. In regular 3D convolution, each convolutional kernel needs to be calculated and accumulated with all layers (input channels) of the input feature map, so the number of input channels for each convolutional kernel is equal to the number of input channels in the input feature map. In depthwise convolution, however, each convolutional kernel is single-channel; one kernel is responsible for one channel, and one channel is convolved by only one convolutional kernel. Therefore, depthwise convolution is sometimes called 2D convolution, meaning it only slides and accumulates along the H and W dimensions.

[0057] As shown in the figure, the input feature map 410b has dimensions of 12×12×3, meaning it includes three channels, each containing a 12×12 image. This depthwise convolution uses three convolutional kernels 420b, each a single-channel kernel with dimensions, for example, 5×5×1. Each kernel convolves only one channel of the input feature map 410b, producing an 8×8×1 output each time. These outputs are then stacked to create an 8×8×3 image, resulting in an 8×8×3 output feature map 430b. As can be seen from the figure, the depth (number of channels) of the output feature map remains consistent with the input feature map. Figure 1 To.

[0058] Since the input channels are not accumulated in depthwise convolution, the dimensions of the input feature map, convolution kernel, and output feature map can be simplified to three dimensions: C (channels), H (height), and W (width).

[0059] Backpropagation during neural network model training involves calculating the gradients of neurons and weights, as shown below:

[0060]

[0061]

[0062] Where top_diff and bottom_diff are the neuron gradients, W is the weight in this iteration, and ΔW is the weight gradient calculated in this iteration. This is the computation in backpropagation, similar to convolution. Relative to the backpropagation direction, the bottom_diff of the previous layer is the top_diff of the current layer, and the bottom_diff of the current layer is the top_diff of the next layer, thus allowing the error to be propagated layer by layer in reverse.

[0063] In the calculation of formula (2), the operation between top_diff and W is similar to the operation between the input neuron and the weight W, where top_diff is equivalent to the input feature map.

[0064] In the calculation of formula (3), the operation between top_diff and bottom_data is similar to the depthwise convolution operation, where top_diff is equivalent to the convolution kernel, which slides and accumulates in the XY direction of bottom_data. The operation principle can be found in [reference]. Figure 4b In this computational scenario, the sizes of top_diff and bottom_data are typically quite large. Therefore, this disclosed embodiment also provides an optimization scheme for convolution operations (referred to as inverse depthwise convolution) in this scenario.

[0065] In backpropagation, for a convolutional layer performing conventional 3D convolution operations, the operation in the reverse process can be called a cross-product convolution operation. The embodiments disclosed herein can also provide optimization schemes for this type of convolution operation.

[0066] Figure 4c An example of the cross-product convolution operation principle that can be applied to the embodiments of this disclosure is shown.

[0067] The figure exemplifies the 3D data `top_diff` of size [Ho Wo Co], which can be represented as a 3D rectangle 410c of size Ho × Wo × Co. The figure also shows 3D data `bottom_data` of size [Hi Wi Ci], which can be represented as a 3D rectangle 420c of size Hi × Wi × Ci. Performing a cross product convolution operation between `top_diff` and `bottom_data` yields output data 430c, which is 4D data of size [Co Kh Kw Ci], and can be represented as Co Kh × Kw × Ci 3D rectangles 430c. Figure 4a The comparison shows that, Figure 4c The cross-product convolution is equivalent to the reverse operation of a regular 3D convolution, that is, the convolution kernel is calculated using the output feature map (top_diff) and the input feature map (bottom_data). Figure 4c The N dimension has been omitted.

[0068] Specifically, for each HoWo facet in top_diff, i.e., for each HoWo facet with a Co value, Ci copies are made, resulting in a Ho×Wo×Ci data set 440c. This 440c data set is then subjected to a depthwise convolution operation with bottom_data (see reference). Figure 4b (See the diagram). This means that the Ci direction is not accumulated, resulting in an output of 460c, which is a 3D data of size Kh×Kw×Ci. For each HoWo surface, this copying and depthwise convolution operation is repeated to obtain Co 3D data of size Kh×Kw×Ci, which is equivalent to obtaining a four-dimensional convolution kernel 430c, with a size of Co×Kh×Kw×Ci.

[0069] In this paper, the terms input feature map, input data, neuron, or input neuron are interchangeable; convolution kernel, filter, or weights are interchangeable. Furthermore, the H (height) and Y dimensions are interchangeable, as are the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In the disclosed embodiment, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window. Furthermore, regardless of the type of convolution operation involved, the data involved can be divided into input feature map, convolution kernel, and output feature map. For example, in the inverse operation, top_diff corresponds to the convolution kernel, bottom_data corresponds to the input feature map, and ΔW corresponds to the output feature map.

[0070] Exemplary computing device

[0071] In this disclosed embodiment, a master-slave computing device can be used to perform the above-described convolution operation. Furthermore, different data paths can be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0072] Figure 5 A schematic structural block diagram of a computing device 500 according to an embodiment of this disclosure is shown. It can be understood that this structure can be considered as... Figure 3 The detailed internal structure of the processing module of a single processing core can also be viewed as a refinement of multiple processing cores. Figure 3 The diagram shows a functional partitioning based on the processing core's computational modules. Figure 5 As shown, the computing device 500 of this disclosure embodiment can be configured to perform various types of convolution operations, and may include a main processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520. Sixteen slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will understand that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and this disclosure embodiment is not limited in this respect.

[0073] The master processing circuit and slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections. In different application scenarios, the connection between multiple slave processing circuits can be either a hard connection arranged by hardwired lines or a logical connection configured according to, for example, microinstructions, to form a topology of various slave processing circuit arrays. The embodiments disclosed herein are not limited in this respect. The master processing circuit and slave processing circuits can cooperate with each other to achieve parallel processing.

[0074] To support computational functions, the main processing circuit and the slave processing circuit can include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit is responsible for the core computations of deep learning algorithms, such as matrix multiplication and convolution.

[0075] The processing circuit can, for example, perform intermediate operations on the corresponding data in parallel according to the operation instructions to obtain multiple intermediate results, and then transmit the multiple intermediate results back to the main processing circuit.

[0076] By configuring the computing device 500 into a master-slave structure (e.g., a master-multiple-slave structure, or a multi-master-multiple-slave structure, which is not limited in this disclosure), for forward computation instructions, the data can be split according to the computation instructions, thereby enabling multiple slave processing circuits to perform parallel computation on the computationally intensive parts to improve computation speed, save computation time, and thus reduce power consumption.

[0077] In some embodiments disclosed herein, by utilizing different data paths to transmit input feature maps and weights, multiple reuse methods for input feature maps and weights can be supported, thereby reducing data access during computation and improving processing efficiency.

[0078] Specifically, the computing device 500 may further include a first storage circuit 530 and a second storage circuit 540 for storing data transmitted via different data channels, respectively. Optionally, the first storage circuit 530 and the second storage circuit 540 may be two storage blocks formed by partitioning the same memory, or they may be two independent memories; no specific limitation is made here.

[0079] The first storage circuit 530 can be used to store multicast data, meaning that the data in the first storage circuit will be transmitted to multiple slave processing circuits via a broadcast bus, and these slave processing circuits will receive the same data. It can be understood that broadcasting and multicasting can be implemented via a broadcast bus. Multicast refers to a communication method that transmits a single data set to multiple slave processing circuits; while broadcasting is a communication method that transmits a single data set to all slave processing circuits, and is a special case of multicast. Since both multicast and broadcasting correspond to one-to-many transmission methods, this document does not specifically distinguish between the two; broadcasting and multicast can be collectively referred to as multicast, and those skilled in the art can understand their meaning from the context.

[0080] The second storage circuit 540 can be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0081] By providing a first storage circuit and a second storage circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by multiplexing multicast data among multiple slave processing circuits.

[0082] In some embodiments, one of the input feature map and the convolution kernel can be determined as multicast data and stored in a first storage circuit to transmit the data to multiple scheduled slave processing circuits via broadcast during computation. Correspondingly, the other of the input feature map and the convolution kernel can be determined as distribution data and stored in a second storage circuit. This distribution data can be distributed to the corresponding slave processing circuits before computation.

[0083] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of this disclosure is also shown. As shown, each slave processing circuit 520 may include multiple arithmetic circuits CU 521, a first buffer circuit 522, and a second buffer circuit 523. Four arithmetic circuits CU0 to CU3 are shown in the figure. Those skilled in the art will understand that the number of arithmetic circuits may be more or less, depending on the specific hardware configuration, and the embodiments of this disclosure are not limited in this respect.

[0084] In some embodiments, the first buffer circuit 522 can be used to buffer the weights or input feature maps allocated to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to buffer the input feature maps or weights allocated to the slave processing circuit. Both buffer circuits are used to select data for computation. The data in the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540; correspondingly, the data in the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computation circuit CU 521 or broadcast to all CUs 521 within the slave processing circuit 520 during computation.

[0085] Each arithmetic circuit CU 521 is used to perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit and data rows selected from the second buffer circuit, respectively, in each arithmetic cycle.

[0086] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by reusing data as much as possible among multiple processing circuits within a single slave processing circuit.

[0087] The processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each arithmetic circuit CU 521.

[0088] Understandable, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but depending on the configuration, the storage circuits and processing circuits can also be combined into a single module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, while the second storage circuit 540 can be shared by multiple slave processing circuits 520, with each slave processing circuit allocated an independent storage area to accelerate access. This disclosure does not limit the embodiments in this respect. Furthermore, in this computing device, the main processing circuit and slave processing circuits can belong to different modules of the same processor or chip, or they can belong to different processors; this disclosure also does not limit this in this respect.

[0089] Exemplary data splitting and storage

[0090] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is usually allocated in contiguous storage space, meaning that multidimensional data can be unfolded one-dimensionally and stored sequentially in memory. For example, in this disclosed embodiment, the initial input feature maps can be stored sequentially in a low-dimensional priority manner (here, C / Ci is the lowest dimension); and to optimize convolution operations, the storage order of the input feature maps can be adjusted during or before the operation, as will be described in detail later. Adjacent dimensions refer to dimensions that are immediately adjacent in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. Adjacent dimensions can also be called continuous dimensions.

[0091] In intelligent processors, due to computational power requirements and considerations of area and power consumption, the main hardware operation unit is the vector multiply-accumulate unit. Implementing support for various convolution algorithms in hardware design essentially involves maximizing the extraction of multiply-accumulate operations from the algorithms and implementing them in on-chip RAM (such as...) via data paths. Figure 3 It efficiently exchanges input and output data for multiplication and addition operations between NRAM, WRAM, etc. and the arithmetic unit.

[0092] Hardware stores data line by line (cache line). Read, write, and computation operations are most efficient when aligned to the entire line. Therefore, to fully utilize bandwidth and accommodate the memory access demands of the arithmetic unit array, data is typically vectorized and aligned. Artificial intelligence chips are usually designed with the Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above, where data along the Ci dimension is continuous. Therefore, vectorization alignment requires the Ci dimension to be aligned to a specified value, such as alignment value M, so that data can be accessed in units of this alignment value M. M can also be called the maximum number of operations per hardware operation. Depending on the hardware design, M can have different values, such as 64 bits, 128 bits, 256 bits, 512 bits, etc. Typically, the size of the arithmetic unit array's input ports is also related to M. For example, in the case of symmetrical input data bit width, the input port size of the arithmetic unit array is usually twice M, meaning it can process input feature map data and weight data of alignment value M in one operation. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0093] When the Ci dimension of the input feature map is small, such as less than the size of a cache line, the Ci dimension needs to be padded to a single line of data (e.g., 512 bits), i.e., filled with invalid data 0. This padding causes a lot of redundant computation, resulting in wasted resources and reduced computational efficiency.

[0094] In this disclosed embodiment, a convolution operation scheme is proposed, which can be, for example, by... Figure 5 The computational device performs this operation. The main processing circuitry acquires the input feature map and / or convolutional kernel. The input feature map and convolutional kernel have been split into multiple split units according to a convolution splitting scheme, and their dimensionality storage order has been transformed so that data within a split unit is stored consecutively as a single data row. Depending on different hardware configurations and / or other considerations, the aforementioned splitting and dimensionality transformation of the input feature map and convolutional kernel can be performed at different locations and times. During the backpropagation of neuron gradient updates, top_diff can be considered as the input feature map.

[0095] In some embodiments, the main processing circuit may include a segmentation circuit, i.e., the segmentation circuit is integrated into the main processing circuit, for splitting and dimensionality-transforming the input feature map and convolutional kernel separately for storage. For example, the main processing circuit can read the input feature map and convolutional kernel in their original storage format from an external storage circuit (e.g., DDR), and then use the segmentation circuit to split and dimensionalize the input feature map and convolutional kernel separately. Afterward, one of the input feature map and convolutional kernel is stored in a first storage circuit, while the other is stored in a second storage circuit. The above-described splitting process can be performed during computation or before computation to prepare the data.

[0096] In other embodiments, the main processing circuit may include a partial segmentation circuitry used only for splitting and dimensionality-transforming data identified as multicast data in the input feature map and convolutional kernels. Data identified as distribution data can be split and dimensionality-transformed via an external segmentation circuitry. For example, in one example, convolutional kernels identified as distribution data can be split and dimensionality-transformed by an external circuitry and pre-stored in a second storage circuitry. This storage can be directly transferred from an off-chip storage circuitry to the second storage circuitry, or it can be transferred via a first storage circuitry to the second storage circuitry.

[0097] In some other embodiments, the main processing circuit may not include the block circuitry at all or may not perform the functions of the block circuitry. In these embodiments, the input feature map and convolutional kernel are split and dimensionally transformed and stored by the block circuitry, which is independent of the main processing circuitry. One of the split and dimensionally transformed input feature map and convolutional kernel may be stored in a first storage circuitry, while the other may be stored in a second storage circuitry.

[0098] The corresponding convolution splitting scheme can be determined based on the minimum storage dimension (e.g., Ci) of the input feature map, where the convolution splitting scheme at least indicates the shape of the splitting unit of the data to be processed. The amount of data contained in a splitting unit does not exceed the maximum hardware computation in a single operation.

[0099] In some embodiments, the amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, so that computational processing can be performed on a unit basis, which can give full play to the hardware's computing power and avoid or reduce invalid computation.

[0100] In the exemplary description disclosed herein, it is assumed that M = 512 bits = 64 bytes, the data type can be Int8, Int16, Float16, or Float32, and the data type of the input feature map is consistent with that of the convolution kernel. Since the data type requires a width of at least 1 byte, and the smallest unit of computation is a single data point, various calculations in the following examples are performed in bytes, such as M = 64 bytes, Ci = 28 bytes, etc. Sometimes, units are omitted for the sake of brevity.

[0101] When the data size of a split unit is equal to M, the shape of each data block in the split unit is blockC*blockY*blockX, which may have several forms. Table 1 lists some of them:

[0102]

[0103] Table 1. Data Block Shapes

[0104] As can be seen from Table 1, some data block shapes have equal X and Y dimensions (as shown in the darker rows), and this shape can simplify subsequent calculations. Therefore, in this disclosed embodiment, this data block shape can be preferably used to split the data to be calculated.

[0105] For simplicity, the 64B×1×1 partitioning scheme is called Forward64, the 16B×2×2 partitioning scheme is called Forward16, the 4B×4×4 partitioning scheme is called Forward4, the 4B×4×4 partitioning scheme applied to depthwise convolution is called Forward1, the 4B×4×4 partitioning scheme applied to inverse depthwise convolution is called Update1, and the 4B×4×4 partitioning scheme applied to cross-product convolution is called Update4. Except for Forward64, these partitioning schemes are suitable for scenarios where the channel C in convolution computation is relatively small, and therefore can be collectively referred to as small convolutions. In these small convolution partitioning schemes, a partitioning unit includes data from the lowest storage dimension and at least one other storage dimension, and the total data volume of a partitioning unit does not exceed the maximum hardware computation in a single operation.

[0106] Different convolution splitting schemes can be applied to different computational scenarios, thereby achieving varying degrees of performance optimization. Specifically, in some embodiments, the corresponding convolution splitting scheme can be determined according to at least one of the following rules:

[0107] Align the lowest storage dimension Ci of the input feature map before splitting to the nearest M / 4. n A multiple of M, where M is the maximum number of operations performed by the hardware in a single operation. And the size Uci (i.e., block C) of the split unit in the lowest storage dimension is determined to be M / 4. n ;

[0108] There are multiple nearest M / 4 n When the value is a multiple of M, take M / 4. n The maximum value is taken as Uci, or the minimum alignment padding value of M / 4 is taken. n As Uci; and

[0109] Determine the size of the split unit in the X and Y storage dimensions, Ux (i.e., blockX) and Uy (blockY), such that Uci × Uy × Ux = M, where preferably Ux = Uy.

[0110] The following examples illustrate the application of the above rules. In all examples, it is assumed that M = 64, then M / 4 n It can be 64, 16, or 4.

[0111] In one example, assuming Ci = 28, then it is aligned to the nearest M / 4. n The multiple is 4*7, so the size Uc (i.e., block C) of the split unit in the lowest storage dimension is determined to be 4. When Ux = Uy is preferred, the shape of the split unit can be determined to be 4B×4×4, which is the Forward4 scheme.

[0112] In another example, assuming Ci = 112, if aligned to 64 * 2 = 128, 16 zeros are added; if aligned to 16 * 7 = 112, no zeros are added; if aligned to 4 * 28 = 112, no zeros are added either. In this case, the nearest M / 4... n The multiples are 16 * 7 = 4 * 28 = 112. According to the rules, we can take M / 4. n The maximum value of 16 is taken as Uc. When Ux = Uy is preferred, the shape of the splitting unit can be determined as 16B×2×2, which is the Forward16 scheme.

[0113] After determining the splitting scheme, the input feature map and convolution kernel can be split into multiple corresponding splitting units according to the determined convolution splitting scheme, and their dimensional storage order can be transformed so that the data in a splitting unit is stored continuously as a data row, which facilitates subsequent reading and processing in units of splitting units (data rows).

[0114] In some embodiments, for three-dimensional or four-dimensional neuron or weight data, it is divided into data blocks of size blockC*blockY*blockX (Uc×Uy×Ux), and each data block is stored consecutively on a row, for example, M=64B, so that when reading a row of data, the data of one data block is actually retrieved.

[0115] Specifically, one or more split units can be read from the data to be processed stored in the first dimension storage order, in the order of the first reading, and the read split units can be stored on the corresponding storage circuit. The data in each split unit is stored in the second dimension storage order, and the split units are stored in the third dimension storage order.

[0116] Figure 6 An exemplary data storage order according to an embodiment of this disclosure is shown.

[0117] As shown in the figure, 610 represents the storage method of the four-dimensional tensor to be computed, which contains N three-dimensional sub-tensors, with N in the highest dimension. Therefore, the storage order of the first dimension of the four-dimensional tensor is NHWC. Note that H and Y, W and X are interchangeable in this paper. Each sub-tensor is divided into smaller data blocks or split units, with the number of data blocks in each dimension being C / Y / X, respectively.

[0118] Figure 620 in the middle illustrates the storage method for each sub-tensor. Each data block is stored as a contiguous 64-byte row. The order in which data blocks are read changes the order of the rows. In the example shown, data blocks are read in the order of C, then X, and finally Y (the first read order is YXC). Therefore, the rows are stored in the order Y*X*C, meaning the third-dimensional storage order is either YXC or HWC. In this example, the third-dimensional storage order is the same as the first-dimensional storage order. It is understandable that other read orders can be used, resulting in a different third-dimensional storage order than the first-dimensional storage order; these will not be listed here.

[0119] Figure 630 on the right shows the order within each row, that is, the data order within each data block, with a shape of blockC*blockY*blockX. In this case, the storage order of the second dimension is CYX or CHW. The specific splitting schemes will be described in detail later with examples of various exemplary convolution splitting schemes.

[0120] Exemplary grouping operations and data reuse

[0121] The preceding description of the hardware structure of the computing device and exemplary data partitioning and storage methods of the embodiments disclosed herein provides different data paths for the input feature maps and weights involved in the computation, thereby reducing data access during computation and improving computational efficiency by utilizing different data transmission methods (e.g., broadcasting, multicasting, distribution, etc.). Convolution computation requires each input feature map to be multiplied and added with each Co convolution kernel to output Co output feature maps. However, on-chip space cannot necessarily store convolution kernels and input feature maps of all sizes simultaneously. Therefore, for the hardware, there is a series of operations involving repeatedly loading input feature data or weight data. How to balance repeatedly loading input feature data or weight data will have a certain impact on computational efficiency. In actual computation, to reduce frequent off-chip memory access, different reuse methods can be adopted according to the scale characteristics of the data involved in the computation. In convolution operations, there are two main data reuse methods: convolution kernel reuse and input feature map reuse.

[0122] Depending on the scenario, convolutional kernel reuse can be divided into intra-channel convolutional kernel reuse and inter-batch convolutional kernel reuse. Intra-channel convolutional kernel reuse is for a single output channel, i.e., a single output feature map, in which case there is only one set of convolutional kernels. For each input feature map, multiple convolutional windows can reuse the same convolutional kernels. Inter-batch convolutional kernel reuse is for batch processing, i.e., processing multiple input images simultaneously. Multiple input images are processed using the same set of convolutional kernels, thus allowing kernel reuse.

[0123] Similarly, depending on the reuse scenario, input feature map reuse can be divided into intra-channel input feature map reuse and inter-channel input feature map reuse. Intra-channel input feature map reuse is for a single output channel. For each input feature map, its adjacent convolutional windows can reuse part of the input feature map data. Inter-channel input feature map reuse is for multiple output channels, that is, when there are multiple output feature maps (i.e., multiple sets of convolutional kernels). In this case, the input feature map within a convolutional window can perform convolution operations with multiple sets of convolutional kernels.

[0124] As described above regarding the principles of convolution, the results of operations on the Co dimension (C dimension for depthwise convolution) do not need to be accumulated. Therefore, operations on different Co dimensions can be performed relatively independently on different processing circuits. In scenarios with a small number of input channels, the convolution kernels are generally also small; for example, Kh and Kw are usually single digits, and the sizes of Co and Ci are similar. In these embodiments, the size of the output channel Co dimension of the convolution kernel in a single round of operation typically does not exceed the number of scheduled slave processing circuits. Therefore, the operation of a single Co dimension needs to be completed by one or more slave processing circuits. More generally, even when the Co dimension is large, it can be achieved by splitting the operation into multiple rounds, where the Co dimension processed in each round does not exceed the number of scheduled slave processing circuits. Thus, in one example, the number of rounds required to complete the convolution operation and the number of Co dimensions processed in each round, or the corresponding grouping pattern, can be determined first based on the output channel Co dimension of the convolution kernel and the number of scheduled slave processing circuits Ns.

[0125] When determining the number of operation rounds required to complete the convolution operation, the number of Co values ​​processed in each round may be different, so even for the same Co dimension, there may be multiple allocation methods.

[0126] For example, with Figure 5Taking a computing device with 16 slave processing circuits (SLs) as an example, assuming all slave processing circuits are schedulable, i.e., Ns = 16. When Co = 40, the operation can be divided into three rounds: the first round processes the first 16 Co values, with each SL processing one different Co value; the second round processes the next 16 Co values, with each SL processing one different Co value; and the last round processes the remaining 8 Co values, with every two SLs processing one different Co value. Alternatively, the operation can be divided into two rounds: the first round processes the first 32 Co values, with each SL processing two different Co values; and the last round processes the remaining 8 Co values, with every two SLs processing one different Co value. For example, when Co = 12, the operation can be divided into a single round, with each SL processing one different Co value, while four SLs are idle or performing invalid operations. In another allocation method, the operation can be divided into three rounds, processing four consecutive Co values ​​each time, with every four SLs processing one different Co value, thus utilizing all schedulable slave processing circuits in each round. It is understandable that those skilled in the art can conceive of many other allocation schemes.

[0127] Therefore, regardless of the allocation method, Co may have two allocation scenarios in a single round of computation: multiple slave processing circuits process a single Co value, or a single slave processing circuit processes one or more Co values. Specifically, in a single round of computation processing Nco output channels, each Rs SL constitutes a slave processing circuit group SLB, processing the convolution kernel corresponding to the same output Co value. Rs = [Ns / Nco], meaning the same convolution kernel is reused on Rs SL within the same SLB, where Rs represents the number of times the convolution kernel is reused among slave processing circuits. Correspondingly, the input feature map can be reused among different slave processing circuit groups SLB, Rn = [Ns / Rs], representing the number of times the input feature map is reused among slave processing circuits.

[0128] Optionally or additionally, when each slave processing circuit processes convolutional kernels corresponding to rn Co values, rn = [Nco / Ns], the input feature map processed by each slave processing circuit can be reused for rn convolutional kernels, where rn represents the number of times the input feature map is reused within a single slave processing circuit. Factors such as hardware buffer space limitations (e.g.) can be considered. Figure 5 The size of the first and second buffer circuits in the circuit determines the maximum number of convolution kernel reuses rs and the maximum number of input feature map reuses rn that can be applied within a single processing circuit.

[0129] Considering the limitations of cache size and the benefits of reuse in hardware circuits, in some embodiments disclosed herein, the case where a slave processing circuit processes multiple Co values ​​in a single round of operation is temporarily not considered, but only the case where one or more slave processing circuits process only one Co value in a single round of operation is considered.

[0130] Different grouping patterns can be adopted depending on the number of slave processing circuits (SLs) that process the same Co value in a single round of computation. It is understood that it is preferable to distribute the callable slave processing circuits (SLs) evenly to balance computing power; for example, grouping them into groups of two, so that 16 SLs can process 8 Co values ​​simultaneously; or grouping them into groups of four, so that 16 SLs can process 4 Co values ​​simultaneously; and so on. In some embodiments, for Figure 5 The computing device shown includes Ns = 16 SLs and can select the following grouping modes: Group1 mode, Group4 mode, and Group16 mode. Those skilled in the art will understand that different grouping modes are possible depending on the value of Ns, and each grouping mode can be processed accordingly by referring to the three representative grouping modes given above in this document.

[0131] In some embodiments, the above grouping pattern can be uniformly represented as GroupN, which means that all slave processing circuits (SLs) scheduled in the current round of operation are divided into N groups, each slave processing circuit group (SLB) processes the same Co value, and different slave processing circuit groups (SLBs) process different Co values. In the case of a total of 16 schedulable SLs, N can be 1, 4, or 16, corresponding to Group1, Group4, and Group16 above, respectively.

[0132] Figures 7a-7d Several exemplary grouping patterns according to embodiments of this disclosure are shown. Figure 7a The Group1 pattern is shown. Figure 7b The Group16 pattern is shown. Figure 7c A Group4 pattern is shown, and Figure 7d Another Group4 pattern is shown.

[0133] like Figure 7a As shown, Group1 mode means that all 16 schedulable SLs belong to one group and jointly process a Co value. For example, SL0 to SL15 belong to group G0. Thus, the operation for this output channel is distributed across the 16 SLs. In this mode, it is preferable to broadcast the convolution kernel 720 of the output channel to each SL, while the input feature map 710 is split and distributed to each SL, thereby improving memory access efficiency.

[0134] In one embodiment, the convolution kernel can be stored in Figure 5The input feature map is stored in the first storage circuit 530 for transmission via a broadcast channel. The input feature map can be partitioned according to the XY direction of the output feature map and stored in the second storage circuit 540 for allocation to different SLs. Thus, all SLs jointly compute an output feature map of Co. The partitioning and storage of the input feature map will be described in detail later with reference to the accompanying drawings.

[0135] like Figure 7b As shown, the Group16 mode means that all 16 schedulable input feature maps (SLs) are divided into 16 groups, that is, one SL per group, and each SL processes a different Co value. For example, SL0 belongs to group G0, SL1 belongs to group G1, and so on, until SL15 belongs to group G15. In this mode, the same input feature map 730 can be reused among the 16 SLs, so it is preferable to broadcast the input feature map 730 to each SL, while the convolutional kernels 740 corresponding to different Co values ​​are distributed to the corresponding SLs.

[0136] In one embodiment, the input feature map can be stored in Figure 5 The first storage circuit 530 is used for transmission via a broadcast channel. The convolution kernels are divided according to Co and stored in the second storage circuit 540 for allocation to different SLs. Thus, all SLs compute output feature maps with different Cos for the same input feature map.

[0137] Group 4 mode divides all 16 schedulable SLs into 4 groups, with each group processing one Co value. Each SL group (SLB) includes SLs equal to Rs = Ns / 4 = 4. For example, SL0–SL3 belong to group G0, SL4–SL7 to group G1, SL8–SL11 to group G2, and SL12–SL15 to group G3. This mode is between Group 1 and Group 16, allowing either the convolutional kernel or the input feature map to be designated as multicast data, while the other is designated as distributed data.

[0138] In one embodiment, the convolutional kernels can be divided into 4 groups according to Co and stored in... Figure 5 The first storage circuit 530 transmits the data using a broadcast channel. The input feature map can be divided into four parts along the XY direction of the output feature map and copied four times, stored in the second storage circuit 540, and then distributed to the four SLBs. Each SLB receives the same input feature map, which is then further divided into four parts and distributed to its four SLBs. Thus, all SLBs in each SLB jointly compute an output feature map of Co, while the four SLBs each process a different Co.

[0139] In another embodiment, the convolution kernel can be stored in Figure 5The second storage circuit 540 stores the input feature map, while the first storage circuit 530 stores the feature map. The division method is similar to that in the previous embodiment.

[0140] In this mode, there are multiple ways to divide the convolution kernel between SLBs.

[0141] Figure 7c A method 770 for allocating the Co dimension of a convolutional kernel is illustrated. In this method, the convolutional kernel is divided into four groups, with each group assigned a Co dimension in increments of 1. For example, when Co = 12, the four groups Co are {0, 4, 8}, {1, 5, 9}, {2, 6, 10}, and {3, 7, 11}. Each time, one Co dimension from each group is sent. For example, the first time Co = 0 to 3 is sent, with one Co corresponding to one SLB, and the four SLBs within one SLB sharing the same weights; the second time Co = 4 to 7 is sent, and so on. Therefore, after each round of computation, the Co dimension of the computation results output by each SLB is continuous.

[0142] Figure 7d Another Co allocation method 780 for the convolutional kernel is shown. In this method, the convolutional kernel is continuously and evenly divided into 4 groups according to Co. For example, when Co = 12, the 4 groups Co are {0, 1, 2}, {3, 4, 5}, {6, 7, 8}, and {9, 10, 11}. Each time, one Co of each group is sent. For example, the first time Co = 0, 3, 6, 9 is sent. One Co corresponds to one SLB, and the 4 SLBs within one SLB share the same weights. The second time Co = 1, 4, 7, 10 is sent, and so on. Thus, the Co dimension of the output result of each SLB in multiple rounds of operation is continuous.

[0143] Exemplary splitting of input feature map

[0144] As can be seen from the preceding description, when multiple SLs process a single Co value, the input feature map needs to be split among these multiple SLs. For example, the Group1 grouping mode requires splitting the input feature map into 16 parts, while the Group4 grouping mode requires splitting the input feature map into 4 parts.

[0145] To ensure that the split input feature maps can share the same convolution kernel, they can be partitioned according to the Ho / Wo direction of the output feature map, thus mapping back to the partitioning of the input feature map. In some embodiments, the input feature maps can be partitioned among the Rs slave processing circuits SL included in each slave processing circuit group as follows: based on the size of the corresponding output feature map, the output feature map is evenly divided into Rs output feature blocks of the same shape in the XY dimension (i.e., the Ho / Wo dimension); and based on the input feature map region required to compute each output feature block, the input feature map is divided into Rs input feature blocks in the XY dimension (i.e., the Hi / Wi dimension) for allocation to the Rs slave processing circuits. It is understood that, depending on the convolution kernel size and convolution stride, the input feature maps corresponding to adjacent output points on the output feature map may overlap.

[0146] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of this disclosure is shown. In this example, the input feature map is divided into 16 parts and assigned to 16 SLs, corresponding to the Group1 pattern.

[0147] In the diagram, 810 represents the output feature map of a single Co, which is divided into 16 identically shaped output feature blocks in a 4×4 pattern along the XY direction, and assigned to SL0 to SL15 respectively. These 16 output feature blocks can then be mapped onto the input feature map 820 to obtain 16 input feature map regions required for calculating these 16 output feature blocks, which are also divided along the XY direction. These 16 input feature map regions can then be correspondingly assigned to 16 slave processing circuits SL.

[0148] As described above, the input feature map is split into units according to a determined convolutional splitting scheme. Therefore, in the above embodiments, the segmentation of the input feature map must ensure that each segmented input feature map block is a multiple of the XY dimension of the splitting unit, meaning it can be aligned with the splitting unit in the XY direction. For example, when choosing a 4×4×4 convolutional splitting scheme, each input feature map block is aligned 4×4; while when choosing a 16×2×2 convolutional splitting scheme, each input feature map block is aligned 2×2.

[0149] If the output feature map is not aligned by the splitting unit (e.g., 4×4 or 2×2), it is necessary to pad the input feature map accordingly (e.g., pad with 0s) so that the actual calculated output XY is aligned by the splitting unit (e.g., 4×4 or 2×2) and the input XY is also aligned by the splitting unit (e.g., 4×4 or 2×2).

[0150] Those skilled in the art will understand that the output feature map can also be split according to other rules in the XY direction, for example, splitting it into 16 identical output feature blocks in a 1×16 manner and assigning them to SL0 to SL15 respectively. The embodiments disclosed herein are not limited in this respect. Furthermore, it is understood that although the foregoing description is in conjunction with the splitting between processing circuits, this splitting method can also be applied to splitting in other scenarios, such as the splitting between arithmetic circuits CU within a single processing circuit SL. The embodiments disclosed herein are not limited in this respect.

[0151] Data storage example on the second storage circuit

[0152] As mentioned earlier, one of the input feature maps or convolutional kernels can be stored in... Figure 5 The data is stored in the first storage circuit 530, and the other of the two can be stored in the second storage circuit 540. Data in the first storage circuit can be multicast via a broadcast path, while data in the second storage circuit is typically distributed. By rationally allocating the storage methods of each data item, the data access speed can be accelerated. In some embodiments, the second storage circuit can allocate a storage area for each slave processing circuit SL, so that the data required for the operation of each slave processing circuit only needs to be read from its corresponding storage area.

[0153] Figures 9a-9d A schematic diagram of data storage in a second storage circuit according to an embodiment of this disclosure is shown. The diagram exemplarily illustrates 16 storage regions 900-915 allocated to, for example, Ns = 16 slave processing circuits SL0-SL15. Each storage region stores the convolution kernel or input feature map to be processed by that slave processing circuit. It will be understood that the content stored in each storage region may vary depending on the different grouping patterns.

[0154] Figure 9a The diagram illustrates that in Group 1 mode, the input feature map is divided into 16 parts, FB0 to FB15, and stored in various storage regions of the second storage circuit. Each storage region corresponding to a SL stores a continuous two-dimensional region, and these two-dimensional regions are arranged according to, for example... Figure 8 The data is split in a specific way. Within each two-dimensional region, the splitting units described above are stored row by row, meaning one row corresponds to one splitting unit of the input feature map. For example, assuming each split input feature block includes 4 splitting units, or 4 rows of data, then in the storage area 1100 allocated to SL0, the first row (Line01), the second row (Line02), the third row (Line03), and the fourth row (Line04) of the input feature map are stored sequentially. Each row can also be called an input feature row.

[0155] Figure 9bThe diagram illustrates that in Group16 mode, convolutional kernels are partitioned according to Co and stored in various storage regions of the second storage circuit to be allocated to corresponding SLs. Each SL's storage region stores the convolutional kernel assigned to it with a different Co value. For example, as described earlier, there are two Co allocation methods, and correspondingly, two storage methods are also available. Figure 9b One approach is illustrated here, where consecutive Co values ​​are sequentially assigned to each SL in each round of computation. This ensures that the Co dimension of the output results from each SL is continuous after each round. For example, the figure shows that convolutional kernels with Co = 0–15 in the first round are stored sequentially in 16 storage regions 900–915; convolutional kernels with Co = 16–31 in the second round are also stored sequentially in 16 storage regions 900–915, and so on. It can be understood that in Group16 mode, the input feature map may also be stored on a second storage circuit (not shown). In this case, the input feature map does not need to be split; it is directly copied 16 times and stored in different storage regions of the second storage circuit to be assigned to the corresponding SL. Thus, each SL can perform convolution operations on the same input feature map but with different Co values.

[0156] Figure 9c This illustration shows one possible storage configuration in Group4 mode. In this example, the input feature map is split into four parts and copied four times, stored in different storage areas of the second storage circuit. Specifically, each processing circuit group (SLB) processes the same input feature map with convolutional kernels of different Co values; and the four processing lines (SLs) within each SLB process one of the split input feature map blocks. Therefore, the storage content of the storage areas used for the four SLBs is the same; for example, the content in 900–903 is the same as the content in 912–915. Furthermore, within each SLB, the storage areas for different SLs store different split input feature blocks; for example, input feature block FB0 is stored in 900, input feature block FB1 is stored in 901, and so on. The same storage allocation is performed in the storage areas of the other SLBs, which will not be described further.

[0157] Figure 9dThis illustrates another possible storage configuration in the Group4 pattern. In this illustrated example, the convolutional kernels are divided into four groups according to Co, and stored in various storage regions of the second storage circuit. Specifically, the convolutional kernels are divided into groups with an interval of 1 according to Co. For example, when Co = 16, they are sequentially allocated to four SLBs in multiple rounds, where Co = 0 is allocated to G0 {SL0~SL3}, Co = 1 is allocated to G1 {SL4~SL7}, Co = 2 is allocated to G2 {SL8~SL11}, and Co = 3 is allocated to G3 {SL12~SL15}; then, starting from Co = 4, they are sequentially allocated to the four SLBs again. The four SLs within each SLB share the same weights. For example, the same weights are stored in storage regions 900, 901, 902, and 903. Similarly, Co can also be used in a contiguous manner within a single SLB; those skilled in the art can deduce its storage method by referring to the foregoing description, which will not be detailed here.

[0158] A single exemplary convolution operation process within the processing circuit

[0159] After the data to be processed is split and stored accordingly, multiple slave processing circuits can be scheduled to perform convolution operations on the corresponding data rows of the input feature map and the convolution kernel. Then, according to the convolution splitting scheme, the operation results returned by the multiple slave processing circuits can be concatenated to obtain the output feature map of the convolution operation between the input feature map and the convolution kernel. Specifically, multiple computational circuits (CUs) and various buffer circuits in the slave processing circuits can be utilized (see...). Figure 5 The specific convolution operation is performed by a buffer circuit. Depending on the size of the buffer circuit inside the processing circuit and the computing power limitations of the arithmetic circuit, multiple calculations are usually required in each round of operation to complete the required operation.

[0160] In some embodiments, the first buffer circuit can be used to cache the input feature map, which may come from a first storage circuit or a second storage circuit; correspondingly, the second buffer circuit can be used to cache the convolution kernel, which may come from a second storage circuit or a first storage circuit. As mentioned above, performing convolution operations on a unit-by-unit basis (a row of data) can fully utilize the hardware's computing power and avoid or reduce invalid computations. Therefore, each arithmetic circuit CU can perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit (e.g., input feature rows) and data rows selected from the second buffer circuit (e.g., weight rows) respectively during each computation. For simplicity, the following description refers to the processing within a single processing circuit SL; it can be understood that similar processing is performed within other SLs.

[0161] As described above, in typical 3D convolution operations, a single slave processing circuit computes one or a portion of the output feature map corresponding to the same output channel Co across all its processing circuits. Depending on the buffer size of the first and second buffer circuits within the slave processing circuit SL, and the processing capability of the processing circuit CU (e.g., internal registers), the slave processing circuit may not be able to compute all the output feature maps allocated to it in one operation. Therefore, the output feature blocks can be divided based on the single-operation capability of the processing circuit (e.g., computing Nop output points or partial sums in a single operation), with each output feature block corresponding to all schedulable N output points within a single SL. CU The single-operation capability (N) of each arithmetic circuit CU *Nop output points). For example, as mentioned earlier. Figure 5 Taking a single SL comprising 4 CUs as an example, assuming each CU can calculate Nop = 4 output points or a partial sum of output points in a single operation, then a single SL can calculate 4*4 = 16 output points (or partial sums) in a single operation. Therefore, the output feature map can be divided into output feature blocks aligned with 16 output points along the XoYo dimension, and each output feature block can be calculated individually. It is understood that these 16 output points can be in a 4*4 format or a 1*16 format; this disclosed embodiment is not limited in this respect.

[0162] When calculating the output feature block for each partition, further calculations can be performed on these N blocks. CU The output points of the output feature block are divided among the arithmetic circuits to determine the processing objects of each arithmetic circuit. Then, based on the division of output points, N values ​​can be selected from the first buffer circuit using the splitting unit as a sliding window. CU The input feature data rows are distributed to N. CU Each arithmetic circuit selects the corresponding weight data from the second buffer circuit and broadcasts it to N. CU A computational circuit is used to achieve parallel computation of the output points corresponding to multiple sliding windows by reusing weight data. Nk sliding selections are performed, where Nk is determined by the smaller of the convolution kernel size in the X and Y dimensions and the maximum convolution kernel size supported by a single operation of the processing circuit.

[0163] In some embodiments, when performing a three-dimensional convolution operation, the corresponding weight data can be selected as follows: 1 / Nop weight rows are selected from the second buffer circuit according to the sliding method corresponding to that in the first buffer circuit, Nop-1 copies are made to expand them into an expanded weight row, and broadcast to N in the processing circuit. CU An operational circuit.

[0164] At this time, each arithmetic circuit can perform bitwise multiplication and accumulation on a unit of 1 / Nop data rows for an input feature row from the first buffer circuit and an extended weight data row from the second buffer circuit during each sliding number selection period to obtain Nop partial sums; and accumulate the Nk*Nop partial sums calculated during Nk sliding number selection periods according to the corresponding convolution output points to obtain and output Nop operation results.

[0165] When the processing circuit outputs the output points of its internal arithmetic circuits, it can output the output points calculated by multiple arithmetic circuits in a specific order according to the division of the output points, so that the continuously output points are continuous in the X and / or Y dimensions, which facilitates subsequent processing. In some embodiments, the aforementioned block circuit can further store the calculation results returned from each processing circuit in a fourth-dimensional storage order. Depending on the situation, the block circuit can also convert the calculation results into the desired dimensional storage order.

[0166] There are multiple ways to divide the output points between operational circuits, and the sliding selection convolution process and the output order of the output points will also be different accordingly.

[0167] Figures 10a-10b This diagram illustrates two different output point divisions between operational circuits.

[0168] Figure 10a A schematic diagram illustrating the allocation of consecutive output points for each arithmetic circuit according to some embodiments of this disclosure is shown. In these embodiments, N CU The output feature block is divided equally among the arithmetic circuits into N. CU Each output feature sub-block has an identical shape, and each sub-block contains Nop output points. Each arithmetic circuit is responsible for calculating one of these sub-blocks. For example, using the example above, the figure shows output feature block 1010a containing 4*4 output points, and each of the equally divided output feature sub-blocks 1011a to 1011d contains 2*2 output points. Each arithmetic circuit calculates 2*2 consecutive output points (or a partial sum) at a time. The figure uses different backgrounds to show the output points assigned to the four different arithmetic circuits CU0 to CU3.

[0169] Based on the above output point partitioning, when performing convolution operations using sliding number selection, the data required to calculate the output feature sub-blocks can be combined with these N... CU Correspondingly, N output feature sub-blocks are selected from the first buffer circuit. CU The calculation is performed on each row of data.

[0170] For example, when selecting the first number of input feature data, the first input data row can be selected from the corresponding input feature blocks according to the four input feature blocks required to calculate the four output feature sub-blocks 1011a to 1014a, and distributed to the four arithmetic circuits.

[0171] When selecting weight data, the corresponding weight data can be selected from the second buffer circuit and broadcast to N. CU Multiple arithmetic circuits are used to achieve parallel computation of the output points of multiple arithmetic circuits by reusing weight data.

[0172] Furthermore, in some embodiments, in order to fully utilize the computing power inside the computing unit CU (e.g., multiply-accumulate unit), such as calculating Nop output points or partial sums in a single operation, weights can be reused within a single input data line to simultaneously calculate Nop output points or partial sums.

[0173] For example, when selecting weight data, you can take only 1 / Nop weight rows, copy them Nop-1 times to expand them into one weight row. This expanded weight row includes Nop identical 1 / Nop weight rows. The expanded weight row can also be broadcast to N. CU This allows for the reuse of weights across multiple operational circuits, while simultaneously reusing weights at a smaller granularity (e.g., 1 / Nop rows) across the computation of Nop output points of a single operational circuit.

[0174] Therefore, by taking N correspondingly each time CU Take one input feature data row, take 1 / Nop weight rows, copy and expand them into one weight row, and calculate N values ​​each time. CU *Nop output points or partial sums. When the calculation result is a partial sum, the partial sum can be calculated multiple times by sliding the slider. The partial sums of each calculation are accumulated according to their respective output points to obtain the final result.

[0175] Based on the way the output points are divided, the number of slides and the step size of the convolution operation can be determined. According to... Figure 10a The partitioning method uses a sliding step of Nk = Kx * Ky, where Kx and Ky are the smaller of the kernel size in the X and Y dimensions and the maximum kernel size supported by the processing circuit in a single operation, respectively, with a sliding step size of 1. The maximum kernel size supported by the processing circuit in a single operation is determined, for example, by the spatial size of the first and second buffer circuits. It can be understood that when the kernel size exceeds the maximum kernel size, it needs to be split in the Kx and Ky directions according to that maximum kernel size.

[0176] according to Figure 10aThis partitioning method allows for the output of each arithmetic circuit to be continuously calculated in the X and / or Y dimensions, thus enabling the output of each circuit's results sequentially. For example, following the order of the arithmetic circuits, the output of one circuit can be completed at a time, such as 2*2 output points, resulting in a 4*4 output feature block after four consecutive iterations.

[0177] Figure 10b A schematic diagram illustrating the allocation of spaced output points for each operational circuit according to other embodiments of this disclosure is shown. In these embodiments, N CU The output feature block is divided equally among the arithmetic circuits into Nop output feature sub-blocks of the same shape. Each output feature sub-block includes N... CU There are N output points, which are respectively assigned to N... CU Each operation circuit. For example, using the example above, the figure shows that output feature block 1010b includes 4*4 output points, and each of the equally divided output feature sub-blocks 1011b to 1011b includes 2*2 output points. In each output feature sub-block, these 2*2 output points are allocated to 4 operation circuits. Thus, each operation circuit calculates one output point in each of the Nop output feature sub-blocks. The figure uses different backgrounds to show the output points allocated to the 4 different operation circuits CU0 to CU3.

[0178] Based on the above output point division, when performing convolution operations through sliding selection, N can be selected from the first buffer circuit according to the data required to calculate the output feature sub-blocks, corresponding to the output point positions of each output feature sub-block. CU The calculation is performed on each row of data.

[0179] For example, during the initial selection of input feature data, four input data rows can be selected from the four input feature blocks required to calculate the four output points within the first output feature sub-block 1011b, and distributed to the four arithmetic circuits. It can be understood that since these four output points are continuous in the X and / or Y directions, the interval or step size of the four simultaneously selected input data rows in the X and / or Y directions is 1.

[0180] When selecting weight data, the corresponding weight data can be selected from the second buffer circuit and broadcast to N. CU Multiple arithmetic circuits are used to achieve parallel computation of the output points of multiple arithmetic circuits by reusing weight data.

[0181] Furthermore, in some embodiments, in order to fully utilize the computing power inside the computing unit CU (e.g., multiply-accumulate unit), such as calculating Nop output points or partial sums in a single operation, weights can be reused within a single input data line to simultaneously calculate Nop output points or partial sums.

[0182] For example, when selecting weight data, you can take only 1 / Nop weight rows, copy them Nop-1 times to expand them into one weight row. This expanded weight row includes Nop identical 1 / Nop weight rows. The expanded weight row can also be broadcast to N. CU This allows for the reuse of weights across multiple operational circuits, while simultaneously reusing weights at a smaller granularity (e.g., 1 / Nop rows) across the computation of Nop output points of a single operational circuit.

[0183] Therefore, by taking N correspondingly each time CU Take one input feature data row, take 1 / Nop weight rows, copy and expand them into one weight row, and calculate N values ​​each time. CU *Nop output points or partial sums. When the calculation result is a partial sum, the partial sum can be calculated multiple times by sliding the slider. The partial sums of each calculation are accumulated according to their respective output points to obtain the final result.

[0184] Based on the way the output points are divided, the number of slides and the step size of the convolution operation can be determined. According to... Figure 10b The partitioning method uses a sliding step of Nk = ceil(Kx / 2) * ceil(Ky / 2), where Kx and Ky are the smaller of the kernel size in the X and Y dimensions and the maximum kernel size supported by the processing circuit in a single operation, respectively, with a sliding step size of 2. Similarly, the maximum kernel size supported by the processing circuit in a single operation is determined, for example, by the spatial size of the first and second buffer circuits. It can be understood that when the kernel size exceeds the maximum kernel size, it needs to be split in the Kx and Ky directions according to that maximum kernel size.

[0185] according to Figure 10b The partitioning method, since the output points calculated by each arithmetic circuit are spaced out in the X and / or Y dimensions (i.e., discontinuous), requires selecting a portion of the computational results from a portion of the circuits for output each time to ensure the output points are continuous in the X and / or Y dimensions. For example, one line can output 1*4 computational results each time, returning a 4*4 output feature block four times consecutively. In this example, the first line needs to output two results from CU0 and two results from CU1, the second line needs to output two results from CU2 and two results from CU3, and so on. In another example, 2*2 computational results can still be output each time, returning a 4*4 output feature block four times consecutively. In this example, the first output is the first computational result of each of CU0 to CU3, the second output is the second computational result of each of CU0 to CU3, and so on. In another example, computational results can also be output column-wise, which will not be elaborated here.

[0186] Furthermore, considering the use of registers within the arithmetic circuit CU, a single processing circuit can calculate multiple 4x4 regions in the Xo / Yo direction, for example, up to 16 4x4 regions. In this case, weights or neurons can be reused based on the stored content in the second storage circuit, reducing the reading frequency of the second storage circuit. If the calculated result is a partial sum, it is stored in a register within the arithmetic circuit.

[0187] In these embodiments, each slave processing circuit can control the reading method of the weight data row and the input feature map data row according to the weight reuse and / or input feature map reuse method, so as to perform bitwise multiplication and accumulation operations on the entire convolution window of the convolution output point simultaneously through multiple operations to obtain multiple partial sums and results, and accumulate them to obtain the convolution output at the corresponding convolution output point.

[0188] The following describes the detailed computation process when different convolution splitting schemes are used and applied to different types of convolution operations, with reference to specific embodiments.

[0189] Example: Forward16

[0190] In Forward16, the splitting unit has a shape of 16B×2×2, and its operation can be applied to similar convolutional splitting schemes. The size of the splitting unit indicated by these convolutional splitting schemes can be expressed as Uci×Uy×Ux=M, where Uci is the size of the splitting unit in the lowest storage dimension (e.g., Ci dimension) of the input feature map and the initial convolutional kernel, Ux and Uy are the sizes of the splitting unit in the X and Y storage dimensions of the input feature map and the initial convolutional kernel, respectively, and M is the maximum number of operations per hardware operation. In these convolutional splitting schemes, Uci>Ux=Uy>1, Uci=M / 4 n ,

[0191] For example, assuming M = 64, then M / 4 n The possible values ​​are 64, 16, 4, and 1. Following the rule Uci > Ux = Uy > 1, the splitting unit can be a 16B × 2 × 2 shape. When using this convolutional splitting scheme, the Ci dimension of the input feature map and the convolutional kernel needs to be aligned to 16B. For example, when Ci = 40, it can be aligned to 3 * 16 = 48 by padding with zeros, thus splitting according to 16B × 2 × 2, with 3 splitting units in the Ci dimension.

[0192] For example, assuming M = 128, then M / 4 nThe possible values ​​are 128, 32, 8, and 2. Following the rule Uci > Ux = Uy > 1, the splitting units can be 32B×2×2 or 8B×4×4 in shape. When using this convolutional splitting scheme, the Ci dimension of the input feature map and the convolutional kernel needs to be aligned to 32B or 8B. For example, when Ci = 40, it can be aligned to 2*32 = 64 by padding with zeros, or it can be aligned to 5*8 = 40. In this case, no zero padding is needed, so it is preferable to split according to 8B×4×4, which has 5 splitting units in the Ci dimension.

[0193] Therefore, although the convolution operation process is described below with specific examples from Forward16, these operations can also be applied to convolution splitting schemes similar to Forward16.

[0194] Figure 11 A schematic diagram illustrating the splitting and storage of the Forward16 scheme according to an embodiment of this disclosure is shown. For simplicity, the examples in the diagram assume the data type is Int8.

[0195] Figure 1110 shows the original data to be processed (which can be neurons or weights), stored in the HWC order. The figure also shows the original data to be processed divided into four data blocks 1111-1114 according to the splitting unit, each data block including 16×2×2=64 data.

[0196] Figure 1120 illustrates the format of the split data for easier reading. It can be seen that the original data blocks (e.g., 1111-1114) are arranged as a single row along dimension C (e.g., 1121-1124). Within each row, the data is stored in CHW order. For example, for data row 1121, the four data points with C=0 are stored first, followed by the four with C=1, then the four with C=2, and so on, until finally the four with C=15.

[0197] Specifically, for neurons, the data needs to be arranged from [1Hi Wi Ci] as follows:

[0198] [1*Hi / 2*Wi / 2*Ci / 16*(16×2×2)], the shape of this seven-dimensional tensor.

[0199] For the weights, the data needs to be arranged from [Co Kh Kw Ci] as follows:

[0200] [Co*Kh / 2*Kw / 2*Ci / 16*(16×2×2)], the shape of this seven-dimensional tensor.

[0201] When using Figure 5When the computing device shown executes the Forward16 convolution splitting scheme, it can split the input feature map and convolution kernel into multiple corresponding splitting units by a block circuit integrated within the main processing circuit, or a block circuit that is completely or partially independent of the main processing circuit, according to the Forward16 convolution splitting scheme. The block circuit can also transform the dimensional storage order of the input feature map and convolution kernel so that the data within each splitting unit is stored consecutively as a data row. The split and transformed input feature map and / or convolution kernel can be provided to the main processing circuit or slave processing circuits. The main processing circuit can then distribute the obtained data to multiple slave processing circuits for performing convolution operations; and, according to the convolution splitting scheme, concatenate the operation results returned by the multiple slave processing circuits to obtain the output feature map of the convolution operation of the input feature map and convolution kernel. The multiple slave processing circuits can then perform convolution operations based on the data they obtain and return the operation results to the main processing circuit.

[0202] In a Forward16 scenario, Co is typically aligned to 16. In these embodiments, the convolution splitting scheme can also indicate the number of operation rounds L required to perform the convolution operation, where the number of output channels Co processed in each operation round corresponds to the number of slave processing circuits Ns that can be scheduled in that operation round, so that one slave processing circuit can process one Co value.

[0203] Since each slave processing circuit processes a different Co value, the input feature map can be reused among these slave processing circuits. Therefore, in some embodiments, the input feature map can be determined as multicast data, and the multicast data, after being split and having its dimensional storage order transformed, is stored in a first storage circuit for transmission to the scheduled slave processing circuits via a broadcast bus during computation. Correspondingly, the convolutional kernel can be determined as distribution data, and the distribution data, after being split and having its dimensional storage order transformed, is stored in a second storage circuit for distribution to the corresponding slave processing circuit. This distribution data can be distributed to the corresponding slave processing circuit before computation.

[0204] In this example, the convolution kernels with different Co values ​​allocated to each slave processing circuit in each round of computation can be further stored in the storage area allocated to the corresponding slave processing circuit in the second storage circuit. The stored content in the second storage circuit is, for example, referenced... Figure 9b .

[0205] Accordingly, the first buffer circuit can buffer multiple lines of input feature data broadcast from the first storage circuit; while the second buffer circuit can buffer multiple lines of weight data distributed to the convolution kernel of the slave processing circuit from the second storage circuit. Depending on the specific splitting and / or multiplexing method, these lines of data can be distributed to the corresponding arithmetic circuits or broadcast to all arithmetic circuits within the slave processing circuit during the operation. Subsequently, each arithmetic circuit CU can perform bitwise multiplication and accumulation operations in each operation for the input feature data lines selected from the first buffer circuit and the weight data lines selected from the second buffer circuit, respectively.

[0206] When multiple computational circuits (CUs) within a single processing circuit SL jointly process a single Co value, the output point needs to be split among these CUs. In the Forward16 scheme, the output point splitting method among the four computational circuits (CUs) can be found by referring to... Figure 10a That is, during each calculation, each operational circuit calculates multiple output points that are consecutive in the X and / or Y dimensions on the output feature map.

[0207] Figure 12 A schematic diagram of a single operation process in the Forward16 scheme according to an embodiment of this disclosure is shown. In this example, the first buffer circuit 1210 is 3×3×64B in size, meaning it can cache up to 9 rows of data, and the second buffer circuit 1220 is 2×2×64B in size, meaning it can cache up to 4 rows of data. For consistency with the splitting unit, the storage within the buffer circuits in the figure is also shown in units of splitting units.

[0208] The diagram illustrates the calculation process of the first sliding selection. Following the method corresponding to the division of output points, N is selected from the first buffer circuit using the splitting unit as the sliding window. CU Each of the N input feature lines is sent to N. CU The first processing circuit performs calculations; 1 / Nop weight rows are selected from the second buffer circuit according to the sliding method corresponding to that in the first buffer circuit, where Nop is the maximum number of convolution output points that each processing circuit can compute in a single operation. These Nop-1 rows are then expanded into an expanded weight row and broadcast to N processing circuits. CU An operational circuit.

[0209] Specifically, in Figure 5 In the computing device shown, N CU =4, Nop=4. When dividing the output points, the division is based on the fact that each operational circuit calculates an output feature block including 2×2 output points in each calculation.

[0210] As shown in the figure, from the first buffer circuit 1210, one input feature data line is selected from each of the four input feature blocks corresponding to the divided output point at the starting position and sent to the four arithmetic circuits 1240 in the processing circuit SL. From the second buffer circuit 1220, 1 / 4 of a weight data line is selected at the starting position, copied three times to expand it into an expanded weight data line 1230, and broadcast to the four arithmetic circuits 1240 in the SL.

[0211] During each calculation, each arithmetic circuit performs bitwise multiplication and accumulation on a unit of 1 / Nop data rows for an input feature row from the first buffer circuit and an extended weight row from the second buffer circuit, resulting in Nop partial sums.

[0212] As shown in the figure, four arithmetic circuits 1240 perform bitwise multiplication and accumulation operations on the distributed input feature data rows and the broadcast extended weight data rows to obtain the result 1250. Different background colors in 1250 represent results obtained by different arithmetic circuits 1240. It can be seen that during each calculation, one CU calculates one 2×2 partial sum, and the four CUs obtain a total of four 2×2 partial sums, or 4×4.

[0213] Next, numbers are simultaneously selected via sliding in the first and second buffer circuits for the next calculation. Nk sliding selections are performed, where Nk = Kx * Ky, and Kx and Ky are the smaller of the convolution kernel size in the X and Y dimensions, or the maximum kernel size supported by the processing circuit in a single operation under the current convolution splitting mode (i.e., Forward16). Accordingly, the computation circuit sums the Nk * Nop parts calculated during the Nk sliding calculations according to the corresponding convolution output points, obtaining and outputting Nop computation results.

[0214] In some embodiments, in Forward16 mode, the maximum convolution kernel size supported by a single operation from the processing circuit is 3×3.

[0215] Figure 13 A schematic diagram of the sliding convolution process in the Forward16 scheme according to an embodiment of this disclosure is shown. This example uses a 6×6 input feature map, a 3×3 convolution kernel, and a stride of 1, resulting in an output feature map size of 4×4. The input feature map is aligned to 2×2 and divided into nine 16×2×2 (C×H×W) blocks, stored in the first buffer circuit (shown as 1310 in the diagram, omitting the C dimension). The 3×3 convolution kernel needs to be aligned to 4×4, with zeros padded in the alignment area, and stored in the second buffer circuit (shown as 1320 in the diagram, again omitting the C dimension). During each computation, a 1×1 block from the convolution kernel is selected and copied three times, corresponding exactly to a 2×2 block of the input feature map. This copying operation can be implemented in hardware.

[0216] The range of input feature maps and convolution kernel selection in the first and second buffer circuits for each slide is as follows: Figure 13 As shown, there are 9 images in total, representing a total of 9 slides. In the images, block 1310 represents the input feature map in the first buffer circuit, and the four dashed boxes represent the regions selected for distribution to the four CUs; ​​block 1320 represents the convolution kernel in the second buffer circuit, and the dashed box represents the selected 1 / 4 row, which is copied 3 times, expanded into a single row, and then broadcast to the 4 CUs. The number of slides Nk = Kx * Ky = 9.

[0217] In each calculation, each CU performs bitwise multiplication and accumulation on a data line from the first buffer circuit and an extended data line from the second buffer circuit, in units of 1 / 4 data lines, to obtain 4 partial sums; and in the current operation round, it accumulates the Nk partial sums corresponding to the same convolution output point obtained from Nk calculations, to obtain and output 4 operation results.

[0218] Specifically, for Figure 13 In each image, the number of CUs (Convolutional Units) is Ncu = 4. Each CU calculates the partial sum of 4 output points on the output feature map. This partial sum is the result of accumulating the positional multiplications of 1 / 4 of the data rows, meaning each output point is a 16×1×1 (Ci×Y×X) standard convolution. After sliding Nk = Kx*Ky = 9 times, the accumulation in the Y×X direction is completed, ultimately yielding a complete 4×4 (Y×X) output in one SL (Simplified Latitude and Longitude) (e.g., ...). Figure 10a (As shown). For larger convolutional kernels, the splitting operation needs to be performed in the Kx and Ky directions according to the same principle as above.

[0219] Figure 14 A schematic diagram illustrating the accumulation of sliding convolution results in a Forward16 scheme according to an embodiment of this disclosure is shown.

[0220] As shown in Figure 1410, each arithmetic circuit CU performs bitwise multiplication and accumulation on a 1 / 4 data line unit for each input feature data line from the first buffer circuit and an extended weight data line from the second buffer circuit during each calculation, to obtain 4 partial sums.

[0221] In the current computational cycle, each computational circuit (CU) accumulates the sums of the Nk partial sums corresponding to the same convolution output point obtained from Nk = Kx * Ky calculations, resulting in four computational results.

[0222] It is understandable that when Ci > 16, it is necessary to traverse along the Ci direction, switching inputs and weights simultaneously, until the complete output is calculated. When the Xo / Yo calculated by each CU is greater than 4, it is necessary to slide along the Xo / Yo direction to read different input neurons and weights. Those skilled in the art can similarly deduce the calculation process based on the foregoing description, and it will not be repeated here.

[0223] As seen from the preceding sliding convolution process, the output of the sliding mode is not in the normal order of traditional convolution output data. Therefore, during the output process, each slave processing circuit (SL) can convert the operation results of its internal computation circuits (CU) into a specified format, such as Nco×Uy×Ux. In some embodiments, each slave processing circuit can output Nop operation results of one of its internal computation circuits at a time, following the order of consecutive output point divisions. The block circuit can further store the operation results returned from each slave processing circuit in a fourth-dimensional storage order. Depending on the situation, the block circuit can also convert the operation results into the desired dimensional storage order for storage.

[0224] Figure 15 A schematic diagram of the output data format of the Forward16 splitting scheme according to an embodiment of this disclosure is shown.

[0225] Figure 1510 shows the raw output of one SL. As can be seen from the figure, each CU computes a 2×2 output neuron. Since the four output neurons computed by one CU are adjacent, each SL can output the computation result of one CU in a sequential order of output point partitioning, i.e., a 1×2×2 (Co×Y×X) region each time, returning a 1×4×4 (Co×Y×X) region after four consecutive iterations, i.e., the four computation results of each of the four CUs. Different CUs within the same SL output different regions of the output feature map of the same Co. Different SLs output output feature maps of different Cos.

[0226] Figure 1520 shows the data structure for storing 16 SLs. As shown, the output buffer circuit (e.g.) Figure 5 The third buffer circuit can convert the output result into a 16×2×2 format, where 16 corresponds to the number of SL and also to the number of output channels Co.

[0227] In some embodiments, considering the storage space of the internal registers of the arithmetic circuits, for example, a single slave processing circuit containing four arithmetic circuits can calculate up to 16 4×4 output feature regions, so weights can be reused, thereby reducing the reading frequency of the second storage circuit. That is, the reading frequencies of the first and second storage circuits can be different. If the result calculated by the arithmetic circuit is a partial sum, it is stored in the register.

[0228] In these embodiments, the processing circuit can be further configured to: determine the number of weight reuses rs within the processing circuit based on storage space limitations within the arithmetic circuit; and control the loading frequency of input feature data in the first buffer circuit, such that the weight data loaded in the second buffer circuit is reused rs times each time, and a convolution operation is performed with the corresponding input feature data loaded rs times in the first buffer circuit. In some examples, rs can take a value no greater than 16.

[0229] The above description and explanation of the convolution optimization scheme provided in this disclosure, in conjunction with the specific convolution splitting scheme of Forward16, serve as an example. Based on the teachings of this disclosure, those skilled in the art can conceive of other convolution splitting schemes according to specific hardware circuit configurations (such as the number of processing circuits, the number of arithmetic circuits within the processing circuits, the hardware's single-processing capability, etc.), all of which fall within the scope of this disclosure and will not be listed here.

[0230] This disclosure also provides a method for performing convolution operations using the aforementioned computing device. Those skilled in the art will understand that the steps of performing convolution operations correspond to the various circuits of the computing device described above in conjunction with the accompanying drawings; therefore, the features described above also apply to the method steps and will not be repeated here.

[0231] This disclosure also provides a chip that may include the computing device of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

[0232] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0233] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0234] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0235] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0236] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0237] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A computing device configured to perform a convolution operation, the computing device comprising: The block circuit is used to split the input feature map and convolution kernel into multiple corresponding split units according to the convolution splitting scheme, wherein each split unit includes data of the lowest storage dimension and at least one other storage dimension, and the data volume of a split unit does not exceed the maximum single operation of the computing device; and to transform the dimensional storage order of the input feature map and convolution kernel so that the data in a split unit is stored continuously as a data row, wherein the split and transformed input feature map and / or convolution kernel is provided to the main processing circuit or the slave processing circuit; The main processing circuit is used to distribute the acquired data to multiple slave processing circuits for performing convolution operations; and to concatenate the operation results returned by the multiple slave processing circuits according to the convolution splitting scheme to obtain the output feature map of the input feature map and the convolution operation of the convolution kernel; and The plurality of slave processing circuits are used to perform convolution operations based on the data they obtain, and return the operation results to the main processing circuit. The convolution splitting scheme indicates the shape of the splitting unit; the storage dimension includes channels, height, and width.

2. The computing device of claim 1, wherein the convolution splitting scheme further indicates the number of operation rounds in which the convolution operation is performed, wherein the number of output channels Co processed in each operation round corresponds to the number of schedulable slave processing circuits Ns in that operation round.

3. The computing device according to claim 2, wherein the computing device further comprises a first storage circuit and a second storage circuit. The input feature map is determined to be multicast data. The multicast data, after being split and its dimensional storage order transformed, is stored in a first storage circuit for transmission to multiple scheduled slave processing circuits via a broadcast bus during computation. The convolution kernel is determined as the distribution data. The distribution data, after being split and transformed in dimensional storage order, is stored in the second storage circuit so as to be distributed to the corresponding slave processing circuit before the operation.

4. The computing device according to claim 3, wherein convolution kernels with different Co values ​​allocated to each slave processing circuit in each operation round are respectively stored in the storage area allocated to the corresponding slave processing circuit in the second storage circuit.

5. The computing device according to any one of claims 3-4, wherein each of the slave processing circuits comprises a first buffer circuit, a second buffer circuit, and a plurality of arithmetic circuits, wherein: The first buffer circuit is used to buffer multiple lines of input feature data that are broadcast from the first storage circuit; The second buffer circuit is used to cache multiple weight data rows from the second storage circuit that are distributed to the convolution kernel of the slave processing circuit; and Each arithmetic circuit is used to perform bitwise multiplication and accumulation operations in each operation on the input feature data line selected from the first buffer circuit and the weight data line selected from the second buffer circuit, respectively.

6. The computing device of claim 5, wherein the slave processing circuitry is further configured to operate according to the following in its schedulable N... CU Divide the output points among the operational circuits: During each calculation, each arithmetic circuit calculates multiple output points that are consecutive in the X and / or Y dimensions on the output feature map.

7. The computing device of claim 6, wherein the convolution operation is a three-dimensional convolution operation, and each of the processing circuits is further configured to: According to the method corresponding to the division of the output points, using the splitting unit as a sliding window, N is slidably selected from the first buffer circuit. CU Each of the input feature rows is sent to the N... CU A computational circuit is used for calculation; From the second buffer circuit, select 1 / Nop weight rows according to the sliding method corresponding to that in the first buffer circuit, where Nop is the maximum number of convolution output points that each operation circuit can compute in a single operation. Copy these rows Nop-1 times to expand them into an expanded weight row, and broadcast this expanded weight row to the Nth weight row in the processing circuit. CU Arithmetic circuit; and Perform Nk sliding selections, where Nk = Kx Ky, Kx, and Ky are the smaller of the convolution kernel size in the X and Y dimensions, or the maximum convolution kernel size supported by the processing circuit in a single operation under the convolution splitting mode.

8. The computing device of claim 7, wherein each of the arithmetic circuits is further configured to: In each calculation, for an input feature row from the first buffer circuit and an extended weight row from the second buffer circuit, bitwise multiplication and accumulation are performed in units of 1 / Nop data rows to obtain Nop partial sums; and The Nk calculated in the Nkth sliding calculation The Nop partial sums are accumulated according to the corresponding convolution output points to obtain Nop computation results.

9. The computing device of claim 8, wherein each of the slave processing circuits is further configured to: According to the order of the continuous division of output points, each time outputs Nop operation results of one operation circuit within it.

10. The computing device of claim 5, wherein the slave processing circuitry is further configured to: Based on the storage space limitations within the arithmetic circuit, determine the number of weight reuses rs from the processing circuit; and The loading frequency of input feature data in the first buffer circuit is controlled so that the weight data loaded in the second buffer circuit is reused rs times each time, and convolution operation is performed with the corresponding input feature data loaded rs times in the first buffer circuit.

11. The computing device according to any one of claims 1-4, wherein the convolution splitting scheme indicates that the shape of the splitting unit is Uci×Ux×Uy=M, Uci is the size of the splitting unit in the lowest storage dimension of the input feature map and the initial convolution kernel, Ux and Uy are the sizes of the splitting unit in the X and Y storage dimensions of the input feature map and the initial convolution kernel, respectively, M is the maximum single computation of the computing device, Uci>Ux=Uy>1, Uci=M / 4 n n=1,2,... .

12. The computing device according to claim 11, wherein M = 64 bytes, Uci = 16 bytes, Ux = Uy = 2.

13. The computing device of claim 6, wherein each arithmetic circuit computes an output feature block comprising 2×2 output points during each computation.

14. The computing device according to claim 7, wherein N CU =4, Nop=4.

15. The computing device of claim 7, wherein the maximum convolution kernel size supported by the slave processing circuit in a single operation in the convolution splitting mode is 3×3.

16. A chip comprising a computing device according to any one of claims 1-15.

17. A circuit board comprising the chip according to claim 16.

18. A method for performing a convolution operation, the method utilizing a computing device according to any one of claims 1-15 to perform the convolution operation.

Images