Data processing apparatus, data processing method and related product

Abstract

The present disclosure discloses a data processing apparatus, a data processing method for executing a block instruction by using the data processing apparatus, and related products. The data processing apparatus can be included in a combined processing apparatus as a computing apparatus, and the combined processing apparatus can further include an interface apparatus and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The combined processing apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively, for storing data of the computing apparatus and the other processing apparatuses. The scheme of the present disclosure realizes data split storage in small convolution operation, and improves operation processing efficiency.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing. More specifically, this disclosure relates to a data processing apparatus, a data processing method that uses the data processing apparatus to execute block instructions on data, a chip, and a circuit board. Background Technology

[0002] Currently, deep learning has become an important branch of machine learning and is greatly promoting the development of artificial intelligence (AI). The core technology of deep learning—deep neural networks (DNNs)—has been widely applied in many industries.

[0003] Neural networks are among the most critical technologies in artificial intelligence and deep learning, with Convolutional Neural Networks (CNNs) being the most important type. The most crucial computation in CNNs is the convolution operation within the convolutional layers (Conv layers). The function of convolutional layers is to extract features from input data. Through multiple convolutions, complex features can be extracted to ensure the network has sufficient expressive and generalization capabilities. Neural network models contain numerous convolution operations of various types, and the computational performance of these operations significantly impacts the overall computational performance of the neural network model. When neural network models are applied to different fields, such as speech recognition, machine translation, and image processing, the corresponding input feature maps and the dimensions of the weights may vary. To fully utilize the hardware advantages of deep learning processors, optimization is needed for convolution operations of different scales and types to improve the computational performance of the neural network model. Summary of the Invention

[0004] To address at least one or more of the technical problems mentioned above, this disclosure proposes a data processing apparatus in several aspects. This apparatus, by executing block instructions on data, enables data of various dimensions and sizes to be adapted to the hardware for convolution operations, thereby improving the computational efficiency of convolution operations. The convolution operations in the embodiments of this disclosure can be operations within various neural network models, which can be applied to various fields such as image processing, speech processing, and text processing. These processes may include, but are not limited to, recognition and classification.

[0005] In a first aspect, embodiments of this disclosure provide a data processing apparatus, including a control circuit, a first storage circuit, and a second storage circuit, wherein: the first storage circuit is used to store input data before processing; the second storage circuit is used to store output data after processing; and the control circuit is used to determine a preferred alignment value based on the minimum storage dimension of the input data before processing; configure a block instruction based on the preferred alignment value; and execute the block instruction to split the input data stored in the first storage circuit according to a first dimension storage order into split units and store them as output data in the second storage circuit, wherein in the second storage circuit, each split unit is stored in a second dimension storage order, and the split units are stored in a third dimension storage order.

[0006] In a second aspect, embodiments of this disclosure provide a chip that includes the data processing apparatus described in the first aspect.

[0007] In a third aspect, this disclosure provides a board that includes the chip described in the second aspect.

[0008] In a fourth aspect, this disclosure provides a data processing method for executing block instructions on input data using the data processing apparatus of the first aspect described above.

[0009] By using the data processing device, chip, board, and data processing method for executing block instructions by the data processing device as described above, the solution of this disclosed embodiment is optimized for data in various convolution splitting schemes, especially for data with fewer channels, to improve the processing efficiency of block processing, adapt to the processing capability of the hardware computing device, and thus fully utilize the parallel processing capability of multiple slave processing circuits, which can effectively improve the computational efficiency of convolution operation. Attached Figure Description

[0010] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0011] Figure 1 This diagram shows the structure of the board card according to an embodiment of this disclosure;

[0012] Figure 2 This diagram illustrates the structure of the combined processing apparatus according to an embodiment of the present disclosure.

[0013] Figure 3a This diagram illustrates the internal structure of the processor core of a single-core computing device according to an embodiment of the present disclosure.

[0014] Figure 3b This diagram shows a simplified internal structure of a multi-core computing device according to an embodiment of the present disclosure.

[0015] Figure 4 This illustrates an exemplary example of the convolution operation principle that can be applied to the embodiments disclosed herein;

[0016] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0017] Figure 6 An exemplary data storage order according to an embodiment of this disclosure is shown;

[0018] Figures 7a-7c Several exemplary grouping patterns according to embodiments of this disclosure are shown;

[0019] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of this disclosure is shown;

[0020] Figure 9 A schematic diagram illustrating the splitting and storage of the Forward4 scheme according to an embodiment of this disclosure is shown;

[0021] Figure 10 A schematic diagram showing the output point division of the computation circuit in the Forward4 scheme according to an embodiment of this disclosure is shown;

[0022] Figure 11 This diagram illustrates a single operation in the Forward4 scheme according to an embodiment of this disclosure.

[0023] Figure 12 This diagram illustrates sliding convolution in the Forward4 scheme according to an embodiment of this disclosure.

[0024] Figure 13 This diagram illustrates the output data format of the Forward4 scheme according to an embodiment of this disclosure.

[0025] Figure 14 This illustrates the overall data transfer process according to embodiments of this disclosure;

[0026] Figure 15 A schematic concept diagram illustrating Trans Tiling according to an embodiment of this disclosure is shown;

[0027] Figure 16 A schematic diagram showing the front and rear table configurations is provided; and

[0028] Figure 17 This diagram illustrates the execution of block instructions on neuronal data according to an embodiment of this disclosure. Detailed Implementation

[0029] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0031] It should also be understood that the terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0032] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0033] Exemplary hardware environment

[0034] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0035] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0036] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0037] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

[0038] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0039] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0040] Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and / or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, for the purposes of this disclosure only, computing device 201 can be considered to have a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0041] Storage device 204 is used to store data to be processed. It may be DRAM or DDR memory, typically 16G or larger in size, and is used to store data of computing device 201 and / or processing device 203.

[0042] Figure 3a The diagram shows the internal structure of the processing core when the computing device 201 is a single-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31, an arithmetic module 32, and a storage module 33.

[0043] The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

[0044] The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

[0045] The storage module 33 is used to store or move relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. The NRAM 331 stores the input neurons, output neurons, and intermediate results after computation; the WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; the DMA 333 is connected to the DRAM 204 via bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204.

[0046] Figure 3b A simplified schematic diagram of the internal structure of the computing device 201 as a multi-core processor is shown. The multi-core computing device can be abstracted using a hierarchical hardware model. As shown, the multi-core computing device can be abstracted into four levels: Card level 350, Chip level 360, Cluster level 370, and Core level 380. This disclosure primarily concerns the data transmission and computing unit portions of the storage unit; therefore, the accompanying drawings and description briefly illustrate and introduce the relevant computing structure, omitting other parts.

[0047] At the board level, each board contains local DDR memory, and each processor chip serves as a computing and control unit.

[0048] At the chip level, each processor chip contains multiple multiprocessors as computing units.

[0049] At the computing cluster level, each multiprocessor includes multiple accelerator cores as control and computing units, as well as shared SRAM as storage units.

[0050] At the processor core level, each accelerator core contains local memory and an array of local processing units. NFU stands for Neuron Function Unit, used for performing convolution calculations.

[0051] In this multi-core computing device, the storage model includes global memory on the board, SRAM (shared memory) on the cluster, NRAM, WRAM, and registers on the core. To achieve better performance, data movement between storage levels below the card and the balance between memory access and computation can be explicitly controlled. SRAM is contained within the Memory Processing Unit (MPU, or Mem Core). A Core refers to the Intelligent Processing Unit (IPU Core, or Core) in a multi-core computing device. One IPU Core contains NRAM, WRAM, NFU, etc. A Cluster refers to a cluster of processors or computing clusters; typically, a multi-core computing device contains several Clusters, and one Cluster contains one Mem Core + N IPU Cores.

[0052] Exemplary convolution operation types

[0053] Convolutional layers in neural network models perform convolution operations, extracting features by applying convolutional kernels (also called filters, weights, etc.) to the input feature map (also known as input data, neurons, or input neurons). A convolutional layer can contain multiple convolutional kernels, each element of which corresponds to a weight coefficient and a bias. The embodiments disclosed herein can be applied to data splitting for various convolutional operations.

[0054] In conventional 3D convolution operations, assuming the input feature map tensor shape in the convolutional layer is represented by X[N HiWi Ci], the tensor shape of the convolutional kernel is represented by K[Co Kh Kw Ci], and the output is Y[N HoWo Co], then the simplified mathematical formula for convolution operations can be expressed as follows:

[0055] Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0056] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, and sh and sw are the stride in the length and width directions. The formula ignores bias, padding, and dilation, and assumes that the input data X has already been padded and the convolution kernel has already been dilated. The formula ignores the N and C dimensions. The forward computation of the neural network model is independent in the N dimension, and fully connected in the C dimension. When the convolution kernel is working, it scans the input features with a certain stride, performs matrix element-wise multiplication and summation on the input features within the convolution window, and adds the bias.

[0057] Figure 4 An example of a conventional 3D convolution operation principle that can be applied to embodiments of this disclosure is shown.

[0058] The figure exemplifies a four-dimensional input data X of size [N Hi WiCi], which can be represented as N Hi×Wi×Ci 3D rectangles 410. The figure also exemplifies a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co Kh×Kw×Ci 3D convolution kernels 420. The convolution result of the input data X and the convolution kernel K yields the output data Y, which is a four-dimensional data of size [N Ho Wo Co], which can be represented as N Ho×Wo×Co 3D rectangles 430.

[0059] The figure also shows a specific example of a convolution operation, where the input data is a 6×6×3 input feature map 440, omitting the N dimension; the convolution kernel is a 3×3×3 stereo convolution kernel 450, targeting a single Co; and the output data is a 4×4 output feature map 460. The specific operation process is as follows:

[0060] The convolution kernel 450 scans across the input feature map 440 with a certain stride, performs element-wise matrix multiplication on the input features within the convolution window 470, and then adds the bias. That is, the value at each position in the output feature map 460 is obtained by performing a two-dimensional convolution operation between the corresponding block of each input feature map and the corresponding convolution kernel, and then summing the results. For example, the figure shows that the value at position (0,0) on the output feature map 460 (i.e., the convolution output point) is obtained by performing a two-dimensional convolution operation between the convolution window 470 (framed by the black cube in the input feature map) and the 3D convolution kernel 450, resulting in three values, which are then summed to obtain the final value.

[0061] To obtain outputs at other locations, the position of the convolution kernel 450 can be moved on the input feature map 440, which means moving the convolution window of the output point. In the example in the figure, the convolution stride (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the values ​​at positions (0,1) or (1,0) on the output feature map 460 can be obtained respectively.

[0062] As described above, a convolutional layer in a neural network contains N sets of input feature maps, each containing Hi × Wi × Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci × Co convolutional kernels of size Kh × Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output feature map contains Ho × Wo × Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Furthermore, the convolutional operation also involves the convolution stride (Sx, Sy), the size of which affects the size of the output feature map.

[0063] In this paper, the input feature map, input data, neuron, or input neuron are used interchangeably; convolution kernel, filter, or weights are used interchangeably. Furthermore, the H (height) and Y dimensions are used interchangeably, as are the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In the embodiments disclosed herein, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window.

[0064] Exemplary computing device / data processing device

[0065] In this disclosed embodiment, a master-slave computing device can be used to perform the above-described convolution operation. Furthermore, different data paths can be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0066] Figure 5 A schematic structural block diagram of a computing device 500 according to an embodiment of this disclosure is shown. It can be understood that this structure can be considered as... Figure 3a The detailed internal structure of the processing module of a single processing core can also be viewed as a refinement of multiple processing cores. Figure 3a The diagram shows a functional partitioning based on the processing core's computational modules. Figure 5As shown, the computing device 500 of this disclosure embodiment can be configured to perform various types of convolution operations, and may include a main processing circuit (MA) 510 and a plurality of slave processing circuits (SL) 520. Sixteen slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will understand that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and this disclosure embodiment is not limited in this respect.

[0067] The master processing circuit and slave processing circuits, as well as multiple slave processing circuits, can communicate with each other through various connections. In different application scenarios, the connection between multiple slave processing circuits can be either a hard connection arranged by hardwired lines or a logical connection configured according to, for example, microinstructions, to form a topology of various slave processing circuit arrays. The embodiments disclosed herein are not limited in this respect. The master processing circuit and slave processing circuits can cooperate with each other to achieve parallel processing.

[0068] To support computational functions, the main processing circuit and the slave processing circuit can include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit is responsible for the core computations of deep learning algorithms, such as matrix multiplication and convolution.

[0069] The processing circuit can, for example, perform intermediate operations on the corresponding data in parallel according to the operation instructions to obtain multiple intermediate results, and then transmit the multiple intermediate results back to the main processing circuit.

[0070] By configuring the computing device 500 into a master-slave structure (e.g., a master-multiple-slave structure, or a multi-master-multiple-slave structure, which is not limited in this disclosure), for forward computation instructions, the data can be split according to the computation instructions, thereby enabling multiple slave processing circuits to perform parallel computation on the computationally intensive parts to improve computation speed, save computation time, and thus reduce power consumption.

[0071] In some embodiments disclosed herein, by utilizing different data paths to transmit input feature maps and weights, multiple reuse methods for input feature maps and weights can be supported, thereby reducing data access during computation and improving processing efficiency.

[0072] Specifically, the computing device 500 may also include a first storage device 530 and a second storage device 540 for storing data transmitted via different data channels, respectively.

[0073] The first storage circuit 530 can be used to store multicast data, meaning that the data in the first storage circuit will be transmitted to multiple slave processing circuits via a broadcast bus, and these slave processing circuits will receive the same data. It can be understood that broadcasting and multicasting can be implemented via a broadcast bus. Multicast refers to a communication method that transmits a single data set to multiple slave processing circuits; while broadcasting is a communication method that transmits a single data set to all slave processing circuits, and is a special case of multicast. Since both multicast and broadcasting correspond to one-to-many transmission methods, this document does not specifically distinguish between the two; broadcasting and multicast can be collectively referred to as multicast, and those skilled in the art can understand their meaning from the context.

[0074] The second storage circuit 540 can be used to store and distribute data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0075] By providing a first storage circuit and a second storage circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by multiplexing multicast data among multiple slave processing circuits.

[0076] In some embodiments, the main processing circuit may determine one of the input feature map and the convolution kernel as multicast data and store it in a first storage circuit to transmit the data to multiple scheduled slave processing circuits via broadcast during computation. Correspondingly, the main processing circuit may determine the other of the input feature map and the convolution kernel as distribution data and store it in a second storage circuit. This distribution data may be distributed to the corresponding slave processing circuits before computation.

[0077] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to an embodiment of this disclosure is also shown. As shown, each slave processing circuit 520 may include multiple arithmetic circuits CU 521, a first buffer circuit 522, and a second buffer circuit 523. Four arithmetic circuits CU0 to CU3 are shown in the figure. Those skilled in the art will understand that the number of arithmetic circuits may be more or less, depending on the specific hardware configuration, and the embodiments of this disclosure are not limited in this respect.

[0078] In some embodiments, the first buffer circuit 522 can be used to buffer the weights or input feature maps allocated to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to buffer the input feature maps or weights allocated to the slave processing circuit. Both buffer circuits are used to select data for computation. The data in the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540; correspondingly, the data in the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding computation circuit CU 521 or broadcast to all CUs 521 within the slave processing circuit 520 during computation.

[0079] Each arithmetic circuit CU 521 is used to perform bitwise multiplication and accumulation operations on data rows selected from the first buffer circuit and data rows selected from the second buffer circuit, respectively, during each calculation.

[0080] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be processed in different transmission methods, thereby reducing the amount of data access by reusing data as much as possible among multiple processing circuits within a single slave processing circuit.

[0081] The processing circuit 520 may also include a third buffer circuit 524 for buffering the calculation results of each arithmetic circuit CU 521.

[0082] Understandable, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but depending on the configuration, the storage circuits and processing circuits can also be combined into a single module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, while the second storage circuit 540 can be shared by multiple slave processing circuits 520, with each slave processing circuit allocated an independent storage area to accelerate access. This disclosure does not limit the embodiments in this respect. Furthermore, in this computing device, the main processing circuit and slave processing circuits can belong to different modules of the same processor or chip, or they can belong to different processors; this disclosure also does not limit this in this respect.

[0083] Exemplary data splitting and storage

[0084] In this disclosed embodiment, the dimensions of the multidimensional data are represented as (N, H, W, C) or (Co, H, W, Ci), which represent the storage order of the data in memory. It is understood that although multidimensional data has multiple dimensions, because the layout of memory is always one-dimensional, there is a correspondence between the multidimensional data and the storage order in memory. Multidimensional data is usually allocated in contiguous storage space, meaning that multidimensional data can be unfolded one-dimensionally and stored sequentially in memory. For example, in this disclosed embodiment, the initial input feature maps can be stored sequentially in a low-dimensional priority manner (here, C / Ci is the lowest dimension); and to optimize convolution operations, the storage order of the input feature maps can be adjusted during the operation, as will be described in detail later. Adjacent dimensions refer to dimensions that are immediately adjacent in the dimensional information representation of multidimensional data; for example, W and Ci are adjacent. Adjacent dimensions can also be called continuous dimensions.

[0085] In intelligent processors, due to computing power requirements and considerations of area and power consumption, the main hardware operation unit is the vector multiply-accumulate unit. Implementing support for various convolution algorithms in hardware design essentially involves maximizing the extraction of multiply-accumulate operations from these algorithms and efficiently exchanging input and output data for multiply-accumulate operations between on-chip RAM (such as NRAM and WRAM in Figure 3) and the arithmetic unit via a data path.

[0086] Hardware stores data line by line (cache line). Read, write, and computation operations are most efficient when aligned to the entire line. Therefore, to fully utilize bandwidth and accommodate the memory access demands of the arithmetic unit array, data is typically vectorized and aligned. Artificial intelligence chips are usually designed with the Ci dimension as the lowest dimension, i.e., the NHWC arrangement mentioned above, where data along the Ci dimension is continuous. Therefore, vectorization alignment requires the Ci dimension to be aligned to a specified value, such as alignment value M, so that data can be accessed in units of this alignment value M. M can also be called the maximum number of operations per hardware operation. Depending on the hardware design, M can have different values, such as 64 bits, 128 bits, 256 bits, 512 bits, etc. Typically, the size of the arithmetic unit array's input ports is also related to M. For example, in the case of symmetrical input data bit width, the input port size of the arithmetic unit array is usually twice M, meaning it can process input feature map data and weight data of alignment value M in one operation. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0087] When the Ci dimension of the input feature map is small, such as less than the size of a cache line, the Ci dimension needs to be padded to a single line of data (e.g., 512 bits), i.e., filled with invalid data 0. This padding causes a lot of redundant computation, resulting in wasted resources and reduced computational efficiency.

[0088] In this disclosed embodiment, a convolution operation scheme is proposed, which determines the corresponding convolution splitting scheme based on the size of the lowest storage dimension (e.g., Ci) of the input feature map, wherein the convolution splitting scheme at least indicates the shape of the splitting unit of the data to be operated on. The amount of data contained in a splitting unit does not exceed the maximum hardware single operation.

[0089] In some embodiments, the amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, so that computational processing can be performed on a unit basis, which can give full play to the hardware's computing power and avoid or reduce invalid computation.

[0090] In the exemplary description disclosed herein, it is assumed that M = 512 bits = 64 bytes, the data type can be Int8, Int16, Float16, or Float32, and the data type of the input feature map is consistent with that of the convolution kernel. Since the data type requires a width of at least 1 byte, and the smallest unit of computation is a single data point, various calculations in the following examples are performed in bytes, such as M = 64 bytes, Ci = 28 bytes, etc. Sometimes, units are omitted for the sake of brevity.

[0091] When the data size of a split unit is equal to M, the shape of each data block in the split unit is blockC*blockY*blockX, which may have several forms. Table 1 lists some of them:

[0092]

[0093] Table 1. Data Block Shapes

[0094] As can be seen from Table 1, some data block shapes have equal X and Y dimensions (as shown in the darker rows), and this shape can simplify subsequent calculations. Therefore, in this disclosed embodiment, this data block shape can be preferably used to split the data to be calculated.

[0095] For simplicity, the 64B×1×1 partitioning scheme is called Forward64, the 16B×2×2 partitioning scheme is called Forward16, the 4B×4×4 partitioning scheme is called Forward4, the 4B×4×4 partitioning scheme applied to depthwise convolution is called Forward1, the 4B×4×4 partitioning scheme applied to inverse depthwise convolution is called Update1, and the 4B×4×4 partitioning scheme applied to cross-product convolution is called Update4. Except for Forward64, these partitioning schemes are suitable for scenarios where the channel C in convolution computation is relatively small, and therefore can be collectively referred to as small convolutions. In these small convolution partitioning schemes, a partitioning unit includes data from the lowest storage dimension and at least one other storage dimension, and the total data volume of a partitioning unit does not exceed the maximum hardware computation time in a single operation.

[0096] Different convolution splitting schemes can be applied to different computational scenarios, thereby achieving different degrees of performance optimization.

[0097] After determining the splitting scheme, the input feature map and convolution kernel can be split into multiple corresponding splitting units according to the determined convolution splitting scheme, and their dimensional storage order can be transformed so that the data in a splitting unit is stored continuously as a data row, which facilitates subsequent reading and processing in units of splitting units (data rows).

[0098] In some embodiments, for three-dimensional or four-dimensional neuron or weight data, it is divided into data blocks of size blockC*blockY*blockX (Uc×Uy×Ux), and each data block is stored consecutively on a row, for example, M=64B, so that when reading a row of data, the data of one data block is actually retrieved.

[0099] Specifically, one or more split units can be read from the data to be processed stored in the first dimension storage order, in the order of the first reading, and the read split units can be stored on the corresponding storage circuit. The data in each split unit is stored in the second dimension storage order, and the split units are stored in the third dimension storage order.

[0100] Figure 6 An exemplary data storage order according to an embodiment of this disclosure is shown.

[0101] As shown in the figure, 610 represents the storage method of the four-dimensional tensor to be computed, which contains N three-dimensional sub-tensors, with N in the highest dimension. Therefore, the storage order of the first dimension of the four-dimensional tensor is NHWC. Note that H and Y, W and X are interchangeable in this paper. Each sub-tensor is divided into smaller data blocks or split units, with the number of data blocks in each dimension being C / Y / X, respectively.

[0102] Figure 620 in the middle illustrates the storage method for each sub-tensor. Each data block is stored as a contiguous 64-byte row. The order in which data blocks are read changes the order of the rows. In the example shown, data blocks are read in the order of C, then X, and finally Y (the first read order is YXC). Therefore, the rows are stored in the order Y*X*C, meaning the third-dimensional storage order is either YXC or HWC. In this example, the third-dimensional storage order is the same as the first-dimensional storage order. It is understandable that other read orders can be used, resulting in a different third-dimensional storage order than the first-dimensional storage order; these will not be listed here.

[0103] Figure 630 on the right shows the order within each row, that is, the data order within each data block, with a shape of blockC*blockY*blockX. In this case, the storage order of the second dimension is CYX or CHW.

[0104] Exemplary grouping operations

[0105] Mini-convolutions use a block-based approach, which has the advantage over traditional convolutions in that Ci-direction alignment only requires the blocks to be aligned in the Ci direction. In this scenario with few channels, the weights (co*Kh*kw*ci) are generally small, with Kh and Kw typically being single digits, and co and ci being roughly equal. This is in contrast to the previous... Figure 5 In the described computing / data processing apparatus, the storage space of the second storage circuit (e.g., WRAM 332 in FIG3) is typically larger than that of the first storage circuit (e.g., NRAM 331 in FIG3). Therefore, in order to make full use of the on-chip computing space, most small convolution schemes, such as Forward4, Forward1, etc., employ a scheme that interchanges the storage locations of neurons and weights with those of normal convolution, that is, storing neurons in the second storage circuit WRAM and weights in the first storage circuit NRAM.

[0106] Convolution computation involves multiplying and adding each input feature map with each convolution kernel of size Co, resulting in Co output feature maps. However, on-chip space cannot always store convolution kernels and input feature maps of all sizes simultaneously. Therefore, hardware involves a series of operations that repeatedly load input feature data or weight data. Balancing the repeated loading of input feature data or weight data significantly impacts computational efficiency. In practical operations, to reduce frequent off-chip memory accesses, there is a strategy for splitting neurons and weights. In some embodiments, different splitting methods can be adopted based on the scale and characteristics of the data involved in the computation.

[0107] As described above regarding the principles of convolution, the results of operations on the Co dimension (C dimension for depthwise convolution) do not need to be accumulated. Therefore, operations on different Co dimensions can be performed relatively independently on different processing circuits. In small convolution scenarios, the size of the Co dimension of the convolution kernel's output channel in a single round of operation typically does not exceed the number of scheduled slave processing circuits. Therefore, the operation of a single Co dimension needs to be completed by one or more slave processing circuits. More generally, even when the Co dimension is large, it can be achieved by splitting the operation into multiple rounds, where the Co dimension processed in each round does not exceed the number of scheduled slave processing circuits. Thus, in one example, the number of rounds required to complete the convolution operation and the number of Co dimensions processed in each round, or the corresponding grouping pattern, can be determined first based on the Co dimension of the convolution kernel's output channel and the number of scheduled slave processing circuits Ns.

[0108] Regardless of the allocation method, in a single round of computation, Co may be allocated in two ways: multiple slave processing circuits process a single Co value, or a single slave processing circuit processes one or more Co values. Specifically, in a single round of computation processing Nco output channels, each Rs SL constitutes a slave processing circuit group SLB, processing the convolution kernel corresponding to the same output Co value. Rs = [Ns / Nco], meaning the same convolution kernel is reused on Rs SL within the same SLB, where Rs represents the number of times the convolution kernel is reused among slave processing circuits. Correspondingly, the input feature map can be reused among the various slave processing circuit groups SLB, Rn = [Ns / Rs], representing the number of times the input feature map is reused among slave processing circuits.

[0109] Optionally or additionally, when each slave processing circuit processes convolutional kernels corresponding to rn Co values, rn = [Nco / Ns], the input feature map processed by each slave processing circuit can be reused for rn convolutional kernels, where rn represents the number of times the input feature map is reused within a single slave processing circuit. Factors such as hardware buffer space limitations (e.g.) can be considered. Figure 5The size of the first and second buffer circuits in the circuit determines the maximum number of convolution kernel reuses rs and the maximum number of input feature map reuses rn that can be applied within a single processing circuit.

[0110] Considering the limitations of cache size and the benefits of reuse in hardware circuits, in some embodiments disclosed herein, the case where a slave processing circuit processes multiple Co values ​​in a single round of operation is temporarily not considered, but only the case where one or more slave processing circuits process only one Co value in a single round of operation is considered.

[0111] Different grouping patterns can be adopted based on the number of slave processing circuits (SLs) that process the same Co value in a single round of computation. It is understood that it is preferable to evenly distribute the callable slave processing circuits (SLs) to balance computing power; for example, grouping them into groups of two, so that 16 SLs can process 8 Co values ​​simultaneously; or grouping them into groups of four, so that 16 SLs can process 4 Co values ​​simultaneously; and so on. (This is in conjunction with the preceding...) Figure 5 In the described computing device, the second storage circuit WRAM has 16 storage areas, each allocated to one of the 16 slave processing circuits SL. Further, every four storage areas can be combined into a storage block, which is then assigned to a corresponding slave processing circuit group SLB. Therefore, in some embodiments, for Figure 5 The computing device shown includes Ns = 16 SLs and can select the following grouping modes: Group1 mode, Group4 mode, and Group16 mode. Those skilled in the art will understand that different grouping modes are possible depending on the value of Ns, and each grouping mode can be processed accordingly by referring to the three representative grouping modes given above in this document.

[0112] In some embodiments, the above grouping pattern can be uniformly represented as GroupN, which means that all slave processing circuits (SLs) scheduled in the current round of operation are divided into N groups, each slave processing circuit group (SLB) processes the same Co value, and different slave processing circuit groups (SLBs) process different Co values. In the case of a total of 16 schedulable SLs, N can be 1, 4, or 16, corresponding to Group1, Group4, and Group16 above, respectively.

[0113] Figures 7a-7c Several exemplary grouping patterns according to embodiments of this disclosure are shown. Figure 7a The Group1 pattern is shown. Figure 7b The Group16 pattern is shown. Figure 7c This demonstrates a Group4 pattern.

[0114] like Figure 7aAs shown, Group1 mode means that all 16 schedulable SLs belong to one group and jointly process a Co value. For example, SL0 to SL15 belong to group G0. Thus, the operation for this output channel is distributed across the 16 SLs. In this mode, it is preferable to broadcast the convolution kernel 720 of the output channel to each SL, while the input feature map 710 is split and distributed to each SL, thereby improving memory access efficiency.

[0115] In one embodiment, the convolution kernel can be stored in Figure 5 The input feature map is stored in the first storage circuit 530 for transmission via a broadcast channel. The input feature map can be partitioned according to the XY direction of the output feature map and stored in the second storage circuit 540 for allocation to different SLs. Thus, all SLs jointly compute an output feature map of Co. The partitioning and storage of the input feature map will be described in detail later with reference to the accompanying drawings.

[0116] like Figure 7b As shown, the Group16 mode means that all 16 schedulable input feature maps (SLs) are divided into 16 groups, that is, one SL per group, and each SL processes a different Co value. For example, SL0 belongs to group G0, SL1 belongs to group G1, and so on, until SL15 belongs to group G15. In this mode, the same input feature map 730 can be reused among the 16 SLs, so it is preferable to broadcast the input feature map 730 to each SL, while the convolutional kernels 740 corresponding to different Co values ​​are distributed to the corresponding SLs.

[0117] In one embodiment, the input feature map can be copied 16 times and stored in 16 storage regions allocated from the processing circuit on a second storage circuit. The convolutional kernels are divided according to Co, with one SL corresponding to one Co. 16 Cos are processed at a time, stored on the first storage circuit, and unicasted to different SLs. Thus, all SLs compute output feature maps with different Cos for the same input feature map.

[0118] like Figure 7c As shown, the Group4 mode divides all 16 schedulable SLs into 4 groups, with each group processing one Co value. Each SL group (SLB) includes SLs equal to Rs = Ns / 4 = 4. For example, SL0–SL3 belong to group G0, SL4–SL7 to group G1, SL8–SL11 to group G2, and SL12–SL15 to group G3. This mode is between Group1 and Group16, thus allowing either the convolutional kernel 760 or the input feature map 750 to be designated as multicast data, while the other is designated as distributed data.

[0119] In one embodiment, the convolutional kernels can be divided into 4 groups according to Co and stored in... Figure 5 The first storage circuit 530 transmits the data using a broadcast channel. The input feature map can be divided into four parts along the XY direction of the output feature map and copied four times, stored in the second storage circuit 540, and then distributed to the four SLBs. Each SLB receives the same input feature map, which is then further divided into four parts and distributed to its four SLBs. Thus, all SLBs in each SLB jointly compute an output feature map of Co, while the four SLBs each process a different Co.

[0120] like Figure 7c As shown, the convolution kernel 760 is divided into 4 groups, with each group further subdivided into groups at intervals of 1 according to Co. For example, when Co = 12, the 4 groups Co 770 are {0, 4, 8}, {1, 5, 9}, {2, 6, 10}, and {3, 7, 11}, respectively. Each time, one Co from each group is sent. For example, the first time Co = 0 to 3 is sent, with one Co corresponding to one SLB, and the 4 SLBs within one SLB sharing the same weights; the second time Co = 4 to 7 is sent, and so on. Therefore, after each round of computation, the Co dimension of the computation results output by each SLB is continuous.

[0121] When using the Forward4 small convolution splitting operation scheme, in order to support the above three modes at the same time, the neurons can be stored in the second storage circuit WRAM and the weights can be stored in the first storage circuit NRAM.

[0122] Exemplary splitting of input feature map

[0123] As can be seen from the preceding description, when multiple SLs process a single Co value, the input feature map needs to be split among these multiple SLs. For example, the Group1 grouping mode requires splitting the input feature map into 16 parts, while the Group4 grouping mode requires splitting the input feature map into 4 parts.

[0124] To ensure that the split input feature maps can share the same convolution kernel, they can be partitioned according to the Ho / Wo direction of the output feature map, thus mapping back to the partitioning of the input feature map. In some embodiments, the input feature maps can be partitioned among the Rs slave processing circuits SL included in each slave processing circuit group as follows: based on the size of the corresponding output feature map, the output feature map is evenly divided into Rs output feature blocks of the same shape in the XY dimension (i.e., the Ho / Wo dimension); and based on the input feature map region required to compute each output feature block, the input feature map is divided into Rs input feature blocks in the XY dimension (i.e., the Hi / Wi dimension) for allocation to the Rs slave processing circuits. It is understood that, depending on the convolution kernel size and convolution stride, the input feature maps corresponding to adjacent output points on the output feature map may overlap.

[0125] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of this disclosure is shown. In this example, the input feature map is divided into 16 parts and assigned to 16 SLs, corresponding to the Group1 pattern.

[0126] In the diagram, 810 represents the output feature map of a single Co, which is divided into 16 identically shaped output feature blocks in a 4×4 pattern along the XY direction, and assigned to SL0 to SL15 respectively. These 16 output feature blocks can then be mapped onto the input feature map 820 to obtain 16 input feature map regions required for calculating these 16 output feature blocks, which are also divided along the XY direction. These 16 input feature map regions can then be correspondingly assigned to 16 slave processing circuits SL.

[0127] As described above, the input feature map is split into units according to a determined convolutional splitting scheme. Therefore, in the above embodiments, the segmentation of the input feature map must ensure that each segmented input feature map block is a multiple of the XY dimension of the splitting unit, meaning it can be aligned with the splitting unit in the XY direction. For example, when choosing a 4×4×4 convolutional splitting scheme, each input feature map block is aligned 4×4; while when choosing a 16×2×2 convolutional splitting scheme, each input feature map block is aligned 2×2.

[0128] If the output feature map is not aligned by the splitting unit (e.g., 4×4 or 2×2), it is necessary to pad the input feature map accordingly (e.g., pad with 0s) so that the actual calculated output XY is aligned by the splitting unit (e.g., 4×4 or 2×2) and the input XY is also aligned by the splitting unit (e.g., 4×4 or 2×2).

[0129] Those skilled in the art will understand that the output feature map can also be split according to other rules in the XY direction, for example, splitting it into 16 identical output feature blocks in a 1×16 manner and assigning them to SL0 to SL15 respectively. The embodiments disclosed herein are not limited in this respect. Furthermore, it is understood that although the foregoing description is in conjunction with the splitting between processing circuits, this splitting method can also be applied to splitting in other scenarios, such as the splitting between arithmetic circuits CU within a single processing circuit SL. The embodiments disclosed herein are not limited in this respect.

[0130] A single exemplary convolution operation process within the processing circuit

[0131] After the data to be processed is split and stored accordingly, multiple slave processing circuits can be scheduled to perform convolution operations on the corresponding data rows of the input feature map and the convolution kernel. Then, according to the convolution splitting scheme, the operation results returned by the multiple slave processing circuits can be concatenated to obtain the output feature map of the convolution operation between the input feature map and the convolution kernel. Specifically, multiple computational circuits (CUs) and various buffer circuits in the slave processing circuits can be utilized (see...). Figure 5 The specific convolution operation is performed by a buffer circuit. Depending on the size of the buffer circuit inside the processing circuit and the computing power limitations of the arithmetic circuit, multiple operation cycles are usually required to complete the required operation in each round of operation.

[0132] As described above, in typical 3D convolution operations, a single slave processing circuit computes one or a portion of the output feature map corresponding to the same output channel Co across all its processing circuits. Depending on the buffer size of the first and second buffer circuits within the slave processing circuit SL, and the processing capability of the processing circuit CU (e.g., internal registers), the slave processing circuit may not be able to compute all the output feature maps allocated to it in one operation. Therefore, the output feature blocks can be divided based on the single-operation capability of the processing circuit (e.g., computing Nop output points or partial sums in a single operation), with each output feature block corresponding to all schedulable N output points within a single SL. CU The single-operation capability (N) of each arithmetic circuit CU *Nop output points). For example, as mentioned earlier. Figure 5 Taking a single SL comprising 4 CUs as an example, assuming each CU can calculate Nop = 4 output points or a partial sum of output points in a single operation, then a single SL can calculate 4*4 = 16 output points (or partial sums) in a single operation. Therefore, the output feature map can be divided into output feature blocks aligned with 16 output points along the XoYo dimension, and each output feature block can be calculated individually. It is understood that these 16 output points can be in a 4*4 format or a 1*16 format; this disclosed embodiment is not limited in this respect.

[0133] When calculating the output feature block for each partition, further calculations can be performed on these N blocks. CU The output points of the output feature block are divided among the arithmetic circuits to determine the processing objects of each arithmetic circuit. Then, based on the division of output points, N values ​​can be selected from the first buffer circuit using the splitting unit as a sliding window. CU The input feature data rows are distributed to N. CU Each arithmetic circuit selects the corresponding weight data from the second buffer circuit and broadcasts it to N. CUA computational circuit is used to achieve parallel computation of the output points corresponding to multiple sliding windows by reusing weight data. Nk sliding selections are performed, where Nk is determined by the smaller of the convolution kernel size in the X and Y dimensions and the maximum convolution kernel size supported by the processing circuit in a single operation under the current convolution splitting mode.

[0134] In some embodiments, when performing a conventional 3D convolution operation, the corresponding weight data can be selected as follows: 1 / Nop weight rows are selected from the second buffer circuit according to the sliding method corresponding to that in the first buffer circuit, Nop-1 copies are made to expand them into an expanded weight row, and broadcast to N in the processing circuit. CU An operational circuit.

[0135] At this time, each arithmetic circuit can perform bitwise multiplication and accumulation on a unit of 1 / Nop data rows for an input feature row from the first buffer circuit and an extended weight data row from the second buffer circuit during each sliding number selection calculation to obtain Nop partial sums; and accumulate the Nk*Nop partial sums obtained from the Nk sliding number selection calculations according to the corresponding convolution output points to obtain and output Nop operation results.

[0136] When a slave processing circuit outputs the output points of its internal arithmetic circuits, it can output the output points calculated by multiple arithmetic circuits in a specific order according to the division of output points, so that the continuously output points are continuous in the X and / or Y dimensions, facilitating subsequent processing. In some embodiments, the main processing circuit can further store the calculation results returned from each slave processing circuit in a fourth-dimensional storage order. Depending on the situation, the main processing circuit can also convert the calculation results into the desired dimensional storage order.

[0137] There are multiple ways to divide the output points between operational circuits, and the sliding selection convolution process and the output order of the output points will also be different accordingly.

[0138] The following section describes in detail the entire process of data splitting, storage, convolution sliding, and computation output using the Forward4 scheme.

[0139] Shape description of input neurons and weights in the Forward4 scheme

[0140] In Forward4, the shape of a splitting unit block is 4B×4×4. The block shape varies slightly depending on the data type. Table 2 shows the block shapes in Forward4 for different data types.

[0141]

[0142] Table 2. Data block shapes in Forward4 under different data types

[0143] Figure 9 A schematic diagram illustrating the splitting and storage of the Forward4 scheme according to an embodiment of this disclosure is shown. For simplicity, the example in the figure assumes the data type is Int8.

[0144] Figure 910 shows the original data to be processed (which can be neurons or weights), stored in the HWC order. The figure also shows four data blocks 911-914 of the original data to be processed, which are divided into splitting units, each containing 4×4×4=64 data points.

[0145] Figure 920 illustrates the format of the split data for easier reading. It can be seen that the original data blocks (e.g., 911-914) are arranged as a single row along dimension C (e.g., 921-924). Within each row, the data is stored in CHW order. For example, for data row 921, the 16 data points with C=0 are stored first, followed by the 16 with C=1, then the 16 with C=2, and finally the 16 with C=3.

[0146] Specifically, for neurons, the data needs to be arranged from [1HiWiCi] as follows:

[0147] [1*Hi / 4*Wi / 4*Ci / 4*(4×4×4)], the shape of this seven-dimensional tensor.

[0148] For the weights, the data needs to be arranged from [Co Kh Kw Ci] as follows:

[0149] [Co*Kh / 4*Kw / 4*Ci / 4*(4×4×4)], the shape of this seven-dimensional tensor.

[0150] As described above, the Forward4 scheme can support multiple grouping patterns. For neurons, the final breakdown of the seven-dimensional shape in the block format into each storage region of the second storage circuit varies slightly depending on the grouping pattern and the HoWo splitting method within the group.

[0151] Assume the original input neuron size is: [1*hi*wi*ci]

[0152] In Group1 grouping mode, the number of input neuron swings varies depending on the HoWo splitting method:

[0153] Ho*Wo 4*4 split: 16[hi / (4*4),wi / (4*4),ci / 4*(4*4*4)]

[0154] Ho*Wo 1*16 split: 16[hi / (4),wi / (4*4*4),ci / 4*(4*4*4)]

[0155] In the above 4*4 split, 16 represents 16 processing circuits SL, and the last 4*4*4 (CHW) represents the CHW block split from three dimensions. In the two divisions of hi and wi, the first 4 means splitting hi*wi into 16 parts and distributing them to 16 SL, and the second 4 means folding hi and wi to the ci direction. The meaning of 1*16 split is the same.

[0156] In Group4 grouping mode, the number of input neuron swings varies depending on the HoWo splitting method:

[0157] Ho*Wo1*4 split: 4*4*[hi / (1*4),wi / (4*4),ci / 4*(4*4*4)]

[0158] For a processing circuit SL: hi / (1*4), wi / (4*4), ci / 4*(4*4*4)

[0159] In the above representation, the first 4 indicates 4 SLBs, and the neuron is replicated 4 times. The second 4 indicates that the neuron is split into 4 SLs of one SLB. The last 4*4*4 indicates that the CHW block is split into three dimensions.

[0160] In the Group16 grouping mode, the input neurons do not need to be split, and their number of rotations is as follows:

[0161] 16*[hi / 4,wi / 4,ci / 4*(4*4*4)]

[0162] The 16 above indicates that the neuron is replicated on 16 SLs, and the 4*4*4 at the end indicates that the CHW is split into three dimensions. The division of hi and wi by 4 indicates that hi and wi are folded to the ci direction.

[0163] Output point splitting between operational circuits in the Forward4 scheme

[0164] When multiple computational circuits (CUs) within a single processing circuit SL process a single Co value, the output point needs to be split among these multiple CUs.

[0165] Figure 10 A schematic diagram illustrating the allocation of spaced output points for each operational circuit in the Forward4 scheme according to some embodiments of this disclosure is shown. In these embodiments, N CU The output feature block is divided equally among the arithmetic circuits into Nop output feature sub-blocks of the same shape. Each output feature sub-block includes N... CUThere are N output points, which are respectively assigned to N... CU Each operation circuit. For example, the figure shows that each SL includes 4 CUs, and each CU can calculate Nop = 4 output points or partial sums in a single operation. Output feature block 1010 includes 4*4 output points, and each output feature sub-block 1011–1014, on average, includes 2*2 output points. Within each output feature sub-block, these 2*2 output points are allocated to the 4 operation circuits. Thus, each operation circuit calculates one output point from each of the 4 output feature sub-blocks. The figure uses different backgrounds to show the output points allocated to the 4 different operation circuits CU0–CU3. It can be seen from the figure that in each calculation, each operation circuit calculates multiple output points spaced in the X and / or Y dimensions on the output feature map.

[0166] Based on the above output point division, when performing convolution operations through sliding selection, N can be selected from the first buffer circuit according to the data required to calculate the output feature sub-blocks, corresponding to the output point positions of each output feature sub-block. CU The calculation is performed on each data row. For example, during the initial selection of input feature data, four input data rows can be selected from the four input feature blocks required to calculate the four output points within the first output feature sub-block 1011, and distributed to the four arithmetic circuits. It can be understood that since these four output points are continuous in the X and / or Y directions, the interval or step size of the four simultaneously selected input data rows in the X and / or Y directions is 1.

[0167] When selecting weight data, the corresponding weight data can be selected from the second buffer circuit and broadcast to N. CU Multiple arithmetic circuits are used to achieve parallel computation of the output points corresponding to multiple arithmetic circuits by reusing weight data. Furthermore, in some embodiments, in order to fully utilize the computing power inside the arithmetic unit CU (e.g., multiply-accumulate unit), for example, to calculate Nop output points or partial sums in a single operation, weights can be reused within a single input data line to simultaneously calculate Nop output points or partial sums.

[0168] For example, when selecting weight data, you can take only 1 / Nop weight rows, copy them Nop-1 times to expand them into one weight row. This expanded weight row includes Nop identical 1 / Nop weight rows. The expanded weight row can also be broadcast to N. CU This allows for the reuse of weights across multiple operational circuits, while simultaneously reusing weights at a smaller granularity (e.g., 1 / Nop rows) across the computation of Nop output points of a single operational circuit.

[0169] Therefore, by taking N correspondingly each time CUTake one input feature data row, take 1 / Nop weight rows, copy and expand them into one weight row, and calculate N values ​​each time. CU *Nop output points or partial sums. When the calculation result is a partial sum, the partial sum can be calculated multiple times by sliding the slider. The partial sums of each calculation are accumulated according to their respective output points to obtain the final result.

[0170] Based on the way the output points are divided, the number of slides and the step size of the convolution operation can be determined. According to... Figure 10 The partitioning method uses a sliding step of Nk = ceil(Kx / 2) * ceil(Ky / 2), where Kx and Ky are the smaller of the kernel size in the X and Y dimensions and the maximum kernel size supported by the processing circuit in a single operation under the current convolution splitting mode, respectively. The sliding step size is 2. The maximum kernel size supported by the processing circuit in a single operation is determined, for example, by at least the spatial size of the first and second buffer circuits. It can be understood that when the kernel size exceeds the maximum kernel size, it needs to be split in the Kx and Ky directions according to that maximum kernel size.

[0171] Convolution sliding process in the Forward4 scheme

[0172] Figure 11 A schematic diagram of a single operation process in the Forward4 scheme according to an embodiment of this disclosure is shown. In this example, the size of the first buffer circuit 1110 is 3×3×64B, meaning it can cache up to 9 rows of data, and the size of the second buffer circuit 1120 is 2×2×64B, meaning it can cache up to 4 rows of data. For consistency with the splitting unit, the storage within the buffer circuits in the figure is also shown in units of splitting units.

[0173] The diagram illustrates the calculation process of the first sliding selection. Following the method corresponding to the division of output points, N is selected from the first buffer circuit using the splitting unit as the sliding window. CU Each of the N input feature lines is sent to N. CU The first processing circuit performs calculations; 1 / Nop weight rows are selected from the second buffer circuit according to the sliding method corresponding to that in the first buffer circuit, where Nop is the maximum number of convolution output points that each processing circuit can compute in a single operation. These Nop-1 rows are then expanded into an expanded weight row and broadcast to N processing circuits. CU An operational circuit.

[0174] Specifically, in Figure 5 In the computing device shown, N CU =4, Nop=4. When dividing the output points, the division is carried out according to the principle that each operation circuit calculates 2×2 output points with a spacing of 1 in both the X and Y dimensions in each calculation.

[0175] As shown in the figure, an input feature data line is selected from the first buffer circuit 1110 at the starting position and at a position moved 1 unit in both the X and / or Y directions, for a total of 4 input feature data lines. These are then sent to the 4 arithmetic circuits 1140 within the processing circuit SL. A quarter of a weight data line (i.e., 2×2 data) is selected from the second buffer circuit 1120 at the starting position, copied 3 times to expand it into an expanded weight data line 1130, and broadcast to the 4 arithmetic circuits 1140 within the SL.

[0176] During each calculation, each arithmetic circuit performs bitwise multiplication and accumulation on a unit of 1 / Nop data rows for an input feature row from the first buffer circuit and an extended weight row from the second buffer circuit, resulting in Nop partial sums.

[0177] As shown in the figure, four arithmetic circuits 1140 perform bitwise multiplication and accumulation operations on the distributed input feature data rows and the broadcast extended weight data rows, obtaining the result 1150. Different background colors in 1150 represent results obtained by different arithmetic circuits 1140. It can be seen that in each operation, one CU calculates the partial sum of four output points, and the four CUs obtain a total of 4×4 partial sums. It can also be seen that the output points calculated by each CU are not adjacent in the XoYo dimensions of the output feature map.

[0178] Next, numbers are simultaneously selected via sliding in the first and second buffer circuits for the next calculation. Nk sliding selections are performed, where Nk = ceil(Kx / 2) * ceil(Ky / 2), where Kx and Ky are the smaller of the convolution kernel dimensions in the X and Y dimensions, or the maximum kernel size supported by the processing circuit in a single operation under the current convolution splitting mode. Correspondingly, the computation circuit sums the Nk*Nop parts calculated in the Nk sliding calculations according to the corresponding convolution output points to obtain Nop computation results.

[0179] In some embodiments, in Forward4 mode, the maximum convolution kernel size supported by a single operation from the processing circuit is 8×8.

[0180] Figure 12A schematic diagram of the sliding convolution process in the Forward4 scheme according to an embodiment of this disclosure is shown. This example uses a 9×9 input feature map, a 5×5 convolution kernel, and a stride of 1, resulting in an output feature map of size 5×5. The input feature map needs to be aligned to 12×12, divided into nine 4×4×4 (C×H×W) blocks, and stored in a first buffer circuit, shown as 1210 in the diagram, where the C dimension is omitted. The 5×5 convolution kernel needs to be aligned to 8×8, with zeros padded in the alignment area, and stored in a second buffer circuit, shown as 1220 in the diagram, again omitting the C dimension. During each computation, a 2×2 block from the convolution kernel is selected and copied four times, corresponding exactly to the 4×4 blocks of the input feature map. This copying operation can be implemented in hardware.

[0181] The range of input feature maps and convolution kernel selection in the first and second buffer circuits for each slide is as follows: Figure 12 As shown, there are 9 images in total, representing a total of 9 slides. In the images, block 1210 represents the input feature map in the first buffer circuit, and the four dashed boxes represent the regions selected to be sent to the four CUs; ​​block 1220 represents the convolution kernel in the second buffer circuit, and the dashed box represents the selected 1 / 4 row, which is copied 3 times, expanded into a row, and then broadcast to the 4 CUs. The number of slides Nk = ceil(Kx / 2) * ceil(Ky / 2) = 9.

[0182] In each calculation, each CU performs bitwise multiplication and accumulation on a 1 / 4 data line unit for an input feature data line from the first buffer circuit and an extended weight data line from the second buffer circuit, to obtain 4 partial sums; and in the current operation round, it accumulates the Nk partial sums corresponding to the same convolution output point obtained from Nk calculations, to obtain and output 4 operation results.

[0183] Specifically, for Figure 12 In each image, the number of CUs (Convolutional Units) is Ncu = 4. Each CU calculates Nop = 4 output points or partial sums in a single operation. This partial sum is the result of accumulating the positional multiplications of 1 / 4 of the data rows, meaning each output point is a 4×2×2 (Ci×Y×X) standard convolution. After sliding Nk = ceil(Kx / 2)*ceil(Ky / 2) = 9 times, the accumulation in the Y×X direction is completed, ultimately yielding a complete 4×4 (Y×X) output in one SL (e.g., ...). Figure 10 (As shown). In this mode, a single computation only supports convolution kernels no larger than 8×8. For larger convolution kernels, they need to be split into 8×8 segments in the Kx and Ky directions, and the splitting operation can be performed according to the same principle as above.

[0184] It is understandable that when Ci > 4, it is necessary to traverse along the Ci direction, switching inputs and weights simultaneously, until the complete output is calculated. When the Xo / Yo calculated by each CU is greater than 4, it is necessary to slide along the Xo / Yo direction to read different input neurons and weights. Those skilled in the art can similarly deduce the calculation process based on the foregoing description, and it will not be repeated here.

[0185] Output shape description in the Forward4 scheme

[0186] As can be seen from the previous output point partitioning method and sliding convolution process, the output results of the sliding mode are not in the normal arrangement order of traditional convolution output data. Therefore, during the output process, each slave processing circuit SL can convert the operation results of its internal operation circuit CU into a specified format, such as Nco×Uy×Ux. In some embodiments, each slave processing circuit can output a portion of the operation results of its internal operation circuits each time, and this portion of the operation results is continuous in the X and / or Y dimensions of the output feature map. The main processing circuit can further store the operation results returned from each slave processing circuit in a fourth-dimensional storage order. Depending on the situation, the main processing circuit can also convert the operation results into the desired dimensional storage order for storage.

[0187] The output data format will vary slightly depending on the grouping mode and / or the splitting method of the input feature map within a single SLB (i.e., the HoWo splitting method based on the output feature map).

[0188] Figure 13 A schematic diagram of the output data format of the Forward4 scheme according to an embodiment of this disclosure is shown. In this embodiment, the grouping mode is Group1, and the input feature map within a single SLB (including 16 SLBs) is split according to Ho×Wo=1×16.

[0189] Figure 1310 shows the raw output of one SL. As can be seen from the figure, each SL outputs a 1×1×4 (Co×Y×X) region at a time, that is, it outputs a portion of the computational results of its internal circuitry each time, for example, two computational results from each of the two CUs (see Figure 1310). Figure 10 This part of the calculation result is continuous in the X and / or Y dimensions of the output feature map, for example, in the same row ( Figure 13 (As shown) or the same column. Returns a 1×4×4 (Co×Y×X) region four times consecutively, representing the four operation results of each of the four CUs. Different SLs output different regions of the output feature map of the same Co. After outputting all 4×4 regions of Co, continuing output will switch between different output points.

[0190] Figure 1320 illustrates the data structure for storing 16 SLs. As shown, the final output data, after being written to the storage circuit (e.g., the first storage circuit), becomes a Yo*Xo*Co*4*16*4 format, where Yo and Xo represent the number of blocks in the output feature map allocated to each SL, and 16 represents the partitioning across the 16 SLs. In some implementations, a slew count operation can be performed again to transform the data into other desired formats as needed.

[0191] As mentioned earlier, the output data format can vary slightly depending on the grouping mode and / or the way the input feature maps are split among multiple SLs within a single SLB. Assume the original output size is:

[0192] 1*ho*wo*co

[0193] Therefore, the output data shape of Group1 when Ho*Wo is split into 4*4 is:

[0194] ho / (4*4)*wo / (4*4)*co / group*(4*16*4)

[0195] In the above formula, (4*16*4) is the basic output block of forward4, with directions corresponding to h*c*w respectively. Here, 16 represents the division of ho and wo with the same co on 16 SLs; ho and wo are divided by two 4s. The first 4 indicates that the data is split into 4×4 when storing data in SL, and the second 4 indicates that the data blocks in the h and w directions are folded. In Group1 mode, the above group = 1.

[0196] The output data shape of Group1 when Ho*Wo is split according to 1*16 is:

[0197] ho / (4)*wo / (4*16)*co / group*(4*16*4)

[0198] In the above formula, (4*16*4) is the basic output block of forward4, with directions corresponding to h*c*w respectively, where 16 represents the division of ho and wo with the same co on 16 SLs; in Group1 mode, the above group = 1. This shape is also... Figure 10 The shape of the diagram.

[0199] Therefore, in Group 1, the 16 input neurons (SLs) evenly divide the Yo*Xo dimension of the output feature map. The data in the inline dimension SL during output corresponds one-to-one with the way the 16 SLs evenly divide the output neurons in the Yo*Xo direction. This scenario is suitable for input neurons with large Y*X direction values ​​and small Co values.

[0200] The output data shape for Group4 is:

[0201] ho / (2*4)*wo / (2*4)*co / group*(4*16*4)

[0202] In the above formula, (4*16*4) has the same meaning as above, except that 16 represents the division of the output of 4 co on 4 SL. In Group4 mode, the above group = 4.

[0203] The output data shape for Group16 is:

[0204] ho / 4*wo / 4*co / group*(4*16*4)

[0205] In the above, (4*16*4) has the same meaning as above, except that 16 represents the output division of 16 co on 16 SL. In Group16 mode, the above group = 16.

[0206] Because there are different splitting categories in the H*W direction within a Group, the 16 in the 4*16*4 mentioned above has different specific splitting methods. Since Forwrd4 uses 4B*4*4 blocks as computational units, alignment constraints are inevitable during computation. Depending on the Group pattern, and even different H*W splitting methods within the same Group pattern, the alignment constraints during computation will ultimately differ. For alignment calculation, the ho*wo alignment constraint can be determined first based on the splitting method of the output feature map, and then the hi*wi constraint can be derived from ho*wo. Since the input neurons need to be arranged in the form of splitting unit blocks, alignment is required again. The above alignment constraints can be summarized in Table 3 below:

[0207]

[0208] Table 3. Alignment Constraints

[0209] In summary, during output, the hardware can automatically output neurons in a 4*16*4 (Y*SL*X) dimension for inline operations and a Y*X*C dimension for interline operations. The same applies to larger convolutional kernels.

[0210] Description of the offset shape in the Forward4 scheme

[0211] Bias is the bias applied after the convolution calculation is complete. The original format of the bias is: [1 1co].

[0212] Since the data output by Forward4 is in the format ho*wo*co / group*(4*16*4), if an offset needs to be added to the data directly output by Forward4 on-chip, the basic shape of the offset needs to be changed. The placement of the offset in on-chip space is related to the Grouping mode. Specifically, the number of offsets in various Grouping modes is as follows:

[0213] In Group1 grouping mode, the number of biased pendulums is: [1 + 1co * 64]

[0214] Where 64 indicates that a single offset is copied 64 times and placed consecutively.

[0215] In Group4 grouping mode, the number of biased pendulums is: [1 + 1co * 16]

[0216] Here, 16 indicates that a single offset is copied 16 times and placed consecutively.

[0217] In Group16 grouping mode, the number of biased pendulums is: [1 + 1co * 4]

[0218] Here, 4 indicates that a single offset is copied 4 times and placed consecutively.

[0219] Data transfer process

[0220] As described above regarding the mini-convolution operation scheme, both the input neurons and weights require dimensionality transformation and storage, and the output neurons also require certain dimensionality transformations. When based on, for example... Figure 3b In the hardware architecture of multi-core computing devices, for hardware I / O efficiency considerations, input data needs to be read from global memory first, and then stored in shared memory (SRAM) after loading. As mentioned earlier, Forward4 requires splitting neurons. Considering alignment factors, its splitting characteristics determine that Forward4 has a greater computational advantage when processing input feature maps with large numbers of channels. Therefore, in the hardware design of Forward4, larger neurons can be stored in WRAM, and relatively smaller weights can be stored in NRAM. At the same time, since the weights and neuron data need to be arranged in the block form described earlier, the neurons stored in WRAM also need to undergo a tensor data shape transformation through NRAM.

[0221] Figure 14 The overall data transfer process according to an embodiment of this disclosure is illustrated.

[0222] As shown in the figure, the weights are read from off-chip storage, such as DDR, into SRAM via the Global Direct Memory Access (GDMA) module. HW face alignment and padding operations are performed on the SRAM. Tiling instructions are used during the data transfer from SRAM to NRAM, which enables both data movement and data dimension transformation and alignment.

[0223] The data transfer process for neurons is similar to that for weights, except that after being transferred to NRAM via block instructions, it also needs to be transferred to WRAM. Because a significant portion of the data overlaps as the convolution kernel slides during neuron computation, this greatly reduces the efficiency of data transfer. To address this issue, some embodiments disclosed herein employ the `img2col` instruction for data distribution, as detailed below.

[0224] Output data can be written back to NRAM, and can also be moved to SRAM by changing the data dimensions using block instructions. Then, it can be written back to off-chip DDR memory via GDMA.

[0225] Exemplary Principles of Blocking Instructions

[0226] Data dimensionality changes and arrangement refer to the process of arranging tensor data of a specific shape into the required specific shape. Data movement refers to the read and write operations performed on data in different memory spaces. As mentioned earlier, the Forward4 convolution operation scheme requires that the neurons and weights used for convolution operations be arranged and aligned according to a specific block pattern. In addition, the output data is also output according to the specific output format of Forward4, which requires that the tensor data be arranged in block form before computation, and also requires that it be arranged back into the normal tensor shape after computation.

[0227] In this disclosed embodiment, the process of moving the input neurons, weights, and bias data from SRAM to NRAM, and the process of moving the output data from NRAM to SRAM, are both accomplished using trans-tiling instructions. This process requires not only the basic data movement but also the dimensional changes and arrangement of the data to meet computational needs.

[0228] The `Deform` instruction family provides the ability to transform data shape and convert data types in the I / O data path, mainly including functions such as `TRANS`, `MOVE`, and `ROTATE`. The mode that implements the transpose function in this instruction family is called `Trans Tiling`, primarily providing performance support for various shape transformations of small convolutions. `Deform` divides a 3D data block into inner and outer layers. The inner layer has three dimensions (corresponding to parameters n0-2 in the instruction), with the lowest dimension's unit being bytes, and the second and highest dimensions being unitless, representing the number of elements in the previous layer. The outer layer also has three dimensions (corresponding to parameters n3-n5 in the instruction), each representing a multiple of the corresponding inner layer dimension.

[0229] When implementing the small convolution splitting scheme, the input data stored in the first dimension storage order (e.g., HWC) needs to be split, dimensionally transformed, and stored in units of splitting units. Within each splitting unit, the data is stored in the second dimension storage order (e.g., CHW), and between splitting units, the data is stored in the third dimension storage order (e.g., HWC).

[0230] Figure 15 A schematic concept diagram of Trans Tiling according to an embodiment of this disclosure is shown.

[0231] The left image shows the input data before deformation. It can be seen that the three-dimensional input data is described using six dimensions: n0 and n3 correspond to the first dimension (e.g., the lowest dimension) of the original three-dimensional data, n1 and n4 correspond to the second dimension (e.g., the second lowest dimension), and n2 and n5 correspond to the third dimension (e.g., the highest dimension) of the data block. In the example in the image, the inner layer of the input data corresponds to the splitting unit. Taking the Forward4 scheme as an example, the inner data block of the input data is a 4B×4×4 data block, where n0 = 4B, n1 = n2 = 4.

[0232] The right side of the figure shows the deformed output data. The three-dimensional output data is also described using six dimensions. At this time, the inner layer of the output data corresponds to the deformed split unit. In the Forward scheme, the inner data block of the output data is a 64B×1×1 data block, where n0=64B, n1=n2=1.

[0233] Furthermore, Trans Tiling also features inline shuffle functionality, including pre-tiling inline shuffle based on pretable and post-tiling inline shuffle based on posttable. Pretable rearrangement rearranges the n0 data at the Tiling input, while posttable rearrangement rearranges the n0 data at the Tiling output. Without considering the table flags, the pretable and posttable are essentially arrays representing the positions of 64 bytes of data.

[0234] Figure 16 A schematic diagram of the front and rear table configuration is shown.

[0235] As shown in the figure, the front and back arrangement tables represent the rearrangement positions of a row of data in dimension n0 of the input or output, which includes 64 bytes. Each byte has 8 bits, including a 6-bit index bit, which records which byte of data from bits 0 to 63 in the original data is stored in this data bit; a 1-bit zero_en bit, which indicates whether to set it to 0. If this bit is 1, it forces a 0, and bits [5,0] are invalid; and a 1-bit mask bit, which indicates whether the data in this bit is valid.

[0236] By using pre- and post-arrangement tables, the data of n0, the input data of block instructions, can be rearranged when needed, and / or the data of n0, the output data of block instructions can be rearranged.

[0237] Table 4 shows the meaning of each parameter of the block instruction. Assuming the bit width of the data to be block is dwidth (in bytes), the size of the data in one atomic operation of the block instruction is called the block bit width T (in bytes). Among the parameters of the block instruction, 11 parameters (n0-n5, s1-s5) are used to describe the tensor shapes of the inner and outer data layers. n0-n2 and s1-s2 describe the inner layer parameters, while n3-n5 and s3-s5 describe the outer layer parameters.

[0238]

[0239]

[0240] Table 4. Parameter meanings of block instructions

[0241] For tensor descriptions before and after block instruction execution, each input tensor requires a set of parameters, described by in0~in5, is1~is5, on0~on5, and os1~os5, totaling 22 parameters. Block instructions support various block bit widths T, such as 1B, 2B, 4B, 6B, 8B, 16B, 32B, etc., with corresponding values ​​set based on different block tasks. Therefore, block instructions also include the block bit width T parameter.

[0242] When using block instructions, there are some basic usage restrictions or constraints, such as: in0, in1, in2, on0, on1, on2 <= 64; n0 requires 64-bit alignment for performance; in0 = on1 * on2 * T, on0 = in1 * in2 * T; in3 * in4 * in5 = on3 * on4 * on5; T <= 32-bit; pre- and post-matching tables = 64-bit.

[0243] Furthermore, block instructions cannot be operated in-place, meaning they require two storage areas. Therefore, in this disclosed embodiment, a data processing apparatus is provided, including a control circuit, a first storage circuit, and a second storage circuit. The first storage circuit stores input data before executing the block instructions; the second storage circuit stores output data after executing the block instructions. The control circuit configures and executes the block instructions. In some embodiments, the data processing apparatus may be, for example, a... Figure 3b The processor cluster shown in the multi-core computing device has a control circuit, such as a processor core within the cluster, a first storage circuit, such as shared SRAM within the cluster, and a second storage circuit, such as NRAM within the processor core. When block instructions are executed for different data (input neurons, weights, output neurons, etc.), the required dimensional changes and transport processes are also different, necessitating the design of different block instruction parameter configuration schemes.

[0244] Block instruction optimization scheme for neurons

[0245] As described above regarding small convolution operation schemes such as Forward4, the role of the block instruction for the input neuron is to split, transform, and store the input neuron stored in the first dimension storage order (e.g., HWC) into split units during the process of moving the input neuron from, for example, SRAM to NRAM. Within each split unit, the input neuron is stored in the second dimension storage order (e.g., CHW), and between split units, the input neuron is stored in the third dimension storage order (e.g., HWC). The shape of the split unit is CHW = U. Ci ×U H ×U W The alignment value M required by the block instruction is U. Ci Multiples of.

[0246] Specifically, for the neuron data in the Forward4 scheme, the data needs to be arranged from [1*hi*wi*ci] as follows:

[0247] [1*hi / 4*wi / 4*ci / 4*(4*4*4)]

[0248] Block instructions have two performance requirements: first, in0 must be based on 64 bytes; second, on0, i.e., in1*in2*T, must also be based on 64 bytes. In other words, both in0 and on0 need to be aligned to an alignment value M = 64 bytes. The shape parameter related to in0 is ci, and the shape parameters related to in1 and in2 are hi*wi.

[0249] Figure 17 A schematic diagram illustrating the execution of block instructions on neuron data according to an embodiment of this disclosure is shown.

[0250] The left image shows the neuron data before block processing (i.e., the input tensor of the block instruction). It can be seen that the three-dimensional neuron data [hi*wi*ci] (N dimensions omitted here) is divided into inner and outer layers, each described using three dimensions. The in0 dimension of the inner data block 1701 is aligned to the baseline alignment value according to the block instruction's constraints, for example, M = 64B; the in1 dimension can be set to U according to the shape of the split unit. W In this example, it is 4; the in2 dimension can also be set to U based on the shape of the splitting unit. H In this example, it is 4. After the inner data block is determined, the sizes of the three outer dimensions in3, in4 and in5 can also be determined accordingly, and their sizes are equal to the number of inner data blocks contained in the corresponding dimension.

[0251] The right side of the figure shows the neuron data after block processing (i.e., the output tensor of the block instruction). It can be seen that the shape of the neuron data now becomes [hi / 4*wi / 4*(ci*16)], which is also divided into inner and outer layers, each described using three dimensions. Since the neuron data needs to be split into blocks by unit, the block width T can be set to U based on the constraints of the block instruction. Ci That is, the amount of data in one atomic operation is U. Ci This facilitates adjusting the storage order by splitting units. In this case, inner data block 1702 can correspond to inner data block 1701 of the input tensor, but its shape is M×U. H ×U W It becomes (M*U) H *U WThe diagram shows a large strip composed of 16 thin strips, each 1×1×1. The on0 dimension of the inner data block is set to in1*in2*T = M, or 64 bytes, according to the block division instruction constraints; the on1 dimension is set to in0 / T = M / T, which is 16 in this example; and the on2 dimension is set to in0 / T / on1 = 1. After determining the inner data block, the sizes of the three outer dimensions on3, on4, and on5 can also be determined accordingly, each equal to the number of inner data blocks containing the output tensor in the corresponding dimension.

[0252] In actual computation, the shape of a neuron is dynamically changing, meaning the size of ci is arbitrary. To accommodate block processing in general cases, the neuron data is divided into integer segments and remainder segments, and block instructions are executed in two segments. While this general approach can support neuron data of any size, it only achieves 100% efficiency when ci is a 64-byte integer segment.

[0253] The small convolution operation scheme is mainly aimed at scenarios with a small number of channels. The main characteristic of this small-channel scenario is that the number of channels is relatively small, usually ci < 64B. For example, the number of channels in the first convolutional layer is generally 4, and for UINT8 type data, ci is only 4B. In this case, if the block processing scheme that supports arbitrary-size neuron data is continued, the efficiency is only 1 / 16 = 6.25%, which is unacceptable in terms of performance. Therefore, it is necessary to propose a more efficient optimization scheme for scenarios where the baseline alignment value M (64B) is smaller than that of the block instruction.

[0254] In small-channel scenarios, the main reason why block division instructions cannot fully utilize their performance is that in0 is "not fully filled," meaning ci is too small. The inventors noticed that in the neuron data shape hi*wi*ci, splitting any adjacent dimensions is equivalent; for example, hi*(wi / 8)*(8*ci) is equivalent to hi*wi*ci. Therefore, it is possible to consider padding adjacent wi-dimensional data into the ci-dimensional block for processing, thereby solving the problem of low performance utilization due to a small in0.

[0255] In some embodiments, the control circuit in the aforementioned data processing apparatus can determine a preferred alignment value P based on the size of the lowest storage dimension (e.g., ci) of the input data before processing; and then configure and execute block instructions based on the preferred alignment value P. It is understood that when the preferred alignment value P is less than the reference alignment value M of the block instructions, it is easier to fill in0, thereby fully utilizing the performance of the block instructions.

[0256] Furthermore, when the preferred alignment value P is less than the baseline alignment value M of the block instruction, the control circuit can treat the input data from its original shape hi*wi*ci as an optimized shape hi*(wi / Ws)*(Ws*ci), where Ws = M / P, representing the data split from the second lowest storage dimension W of the input data by a factor of Ws and then padded to the lowest storage dimension C. hi is the size of the highest storage dimension H of the input data, wi is the size of the second lowest storage dimension W, and ci is the size of the lowest storage dimension C. Thus, by filling the data in the adjacent wi dimension into the ci dimension, the alignment requirement for ci can be reduced.

[0257] Considering the shape of the split unit is CHW=U Ci ×U H ×U W Therefore, in some embodiments, the preferred alignment value P can be selected as U. Ci 2 n times, This facilitates processing from the second-lowest storage dimension W in 2 n Fold the data to the lowest storage dimension Ci.

[0258] In some embodiments, the preferred alignment value P can be determined based on the range of values ​​for ci. Table 5 shows the optimization schemes corresponding to different preferred alignment values ​​P.

[0259]

[0260]

[0261] Table 5. Optimization Scheme for Small Channel Block Processing

[0262] As can be seen from Table 5, the control circuit can determine the preferred alignment value P based on the minimum storage dimension size ci of the input data according to the following rules:

[0263] When 0 <ci≤U Ci At that time, P = U Ci ,

[0264] When 2 n *U Ci <ci≤2n+1*U Ci , When P = 2 n+1 *U Ci ,

[0265] when At that time, P = M.

[0266] Table 5 also shows that the implementation of the block splitting instruction in the small channel scenario is related to the splitting of wi. In Table 5, Ws represents the number of ci parts split from the wi direction to the ci direction (short for wisupply). It should be noted that the smaller the number of channels, the more wi parts are supplied to the ci direction, and the larger the alignment constraint on wi is, which needs to satisfy wi / 4 / Ws>=1.

[0267] Furthermore, although the rules for determining the preferred alignment value P are listed above, these rules are merely preferred embodiments used to select the optimal alignment value that best suits the current ci value. As can be seen from Table 5, optimization schemes with larger ci alignments are compatible with those with smaller ci alignments; that is, schemes with large P values ​​are compatible with schemes with small P values. For example, the ci 64B alignment scheme can actually handle ci of any size, and the ci 32B scheme can actually handle the range ci <= 32B.

[0268] Furthermore, as shown in Table 5, in the optimization scheme, before executing the block division instruction, the data in the wi direction needs to be supplemented to the ci dimension, that is, stored in the ci dimension. At this time, the ci dimension becomes Ws*ci. Therefore, when selecting the splitting unit, if the data is retrieved in the original way, the expected result cannot be obtained, and the data needs to be preprocessed. In order to efficiently abstract and unify the data processing of the four alignment schemes that require wi splitting in the table above, a pre-allocation table is used in some embodiments disclosed in this paper. The role of the pre-allocation table is to change the position of the data in the Ws*ci row, so that the data of the splitting unit data block to be processed is consistent with the data of the splitting unit data block when it was initially not supplemented from wi in terms of dimension storage order (e.g., HWC).

[0269] In these embodiments, the control circuit can set a pre-alignment table for the block instruction based on a preferred alignment value P. This pre-alignment table is used to rearrange the inner lowest dimension data of the input tensor of the block instruction according to the instructions of the pre-alignment table. In one implementation, the control circuit can set the pre-alignment table as follows: the inner lowest dimension data of the input tensor of the block instruction is converted into a shape aligned according to the split unit, and the data of each split unit is arranged according to the storage order of the first dimension.

[0270] Specifically, the pre-allocation table needs to place the data of the wi dimension stored in the ci dimension at the end, while taking into account different wi splitting scenarios. In addition, it also needs to ensure that the data is arranged in the alignment requirements of the splitting units (e.g., the 4*4*4 data block format in the Forward4 scheme). In some embodiments, the pre-allocation table can be configured according to the logic shown in the pseudocode in Table 6 below.

[0271]

[0272]

[0273] Table 6. Pseudocode for tabulation before block instructions in the neuron optimization scheme.

[0274] After the above pre-allocation process, the in0 data of the input tensor of the block instruction is replaced with 4*4*4hwc data in the format of ci first and wi last, which is exactly the same as the case when aligning to the reference alignment value M.

[0275] Based on the above Figure 17 The description of the conventional block instruction execution process for aligning to the baseline alignment value M shows that although the input neurons can be arranged from [1*hi*wi*ci] to the form [1*hi / 4*wi / 4*ci / 4*(4*4*4)], the lowest-dimensional 4*4*4 splitting unit blocks are still in HWC order, not CHW order. To ensure that the lowest-dimensional splitting unit blocks are in CHW order, the post-alignment table described above needs to be used.

[0276] In some embodiments, the control circuitry in the data processing apparatus may be further configured to configure the block instruction as follows: setting a post-allocation table for the block instruction to rearrange the inner lowest-dimensional data of the output tensor of the block instruction according to the instructions of the post-allocation table. Specifically, the control circuitry may be further configured to set the post-allocation table as follows: converting the inner lowest-dimensional on0 data of the output tensor arranged according to a first-dimensional storage order (e.g., HWC) into an arrangement according to a second-dimensional storage order (e.g., CHW).

[0277] At this point, the post-allocation table only retrieves the on0 of the output tensor, which is the writing order of the input in1*in2*T = M = 64B data. This writing order of the 64B data is related to the data bit width dwidth. In some embodiments, the post-allocation table can be configured according to the logic shown in the pseudocode in Table 7 below.

[0278]

[0279] Table 7. Pseudocode for tabulation after neuron block instructions

[0280] Note that in Table 5, in the ci 4B alignment scheme (i.e., P = U), Ci =4B), no pre-alignment table is needed. This is because during 4B alignment, the data supplemented from the wi direction is placed at the end of the ci direction, which does not conflict with the 4B×4×4 splitting unit constraint in the Forward4 scheme. The data format perfectly meets the needs of the post-alignment table, so no data rearrangement through pre-alignment table is required. In other words, it is possible to perform data rearrangement only when P≠U. Ci When P≠M, set the pre-allocation table for the block instruction.

[0281] The inner data blocks are pre-arranged into the desired format (e.g., HWC) by using a pre-allocation table before the block splitting instruction is executed.

[0282] In one embodiment, the parameters of the input tensor of the block instruction can be configured as follows: set the inner lowest dimension size in0 of the input tensor in the block instruction to ci*dwidth*Ws, the inner lowest dimension size in1 to max(1,4 / Ws), and the inner highest dimension size in2 to U. H ; and based on the size of each dimension of the optimized shape of the input data, set the size values ​​in3, in4 and in5 of the three outer dimensions of the input tensor in the block instruction, where the size values ​​of the three outer dimensions represent the number of inner data blocks that are defined by the three inner dimensions of the input tensor in the corresponding dimension.

[0283] Additionally, in one embodiment, the parameters of the output tensor of the block instruction can be configured as follows: the inner lowest dimension size on0 of the output tensor in the block instruction is set to in1*in2*T=M, the inner lowest dimension size on1 is set to in0 / T, and the inner highest dimension size on2 is set to 1; and the size values ​​on3, on4, and on5 of the three outer dimensions of the output tensor in the block instruction are set according to the size of each dimension of the optimized shape of the input data, wherein the size values ​​of the three outer dimensions respectively represent the number of inner data blocks defined by the three inner dimensions of the output tensor in the corresponding dimension.

[0284] In addition to the dimension parameters, the dimension step size also needs to be set. In some embodiments, the step size of adjacent data points in the storage space in the five dimensions other than the lowest inner dimension can be set based on the size of the six dimensions of the input tensor and the output tensor, as well as the size of each dimension of the optimized shape of the input data. In some embodiments, the block width T can be set to max(1,4 / Ws) according to the constraints of the block instruction and the shape transformation of the splitting unit before and after processing.

[0285] In one example, when considering the Forward4 scheme, i.e., the splitting unit is U... Ci ×U H ×U W When M = 4B×4×4 and M = 64B, the block instructions can be configured as shown in Table 8 below.

[0286]

[0287] Table 8. Block Instruction Parameter Configuration Scheme for Neuron Optimization

[0288] Where ci, hi, and wi represent the number of data items in the C, H, and W dimensions of the input data, respectively; dwidth represents the data bit width; B represents bytes; T represents the block bit width; is1 to is5 represent the five-dimensional step size of the input tensor; and os1 to os5 represent the five-dimensional step size of the output tensor.

[0289] Therefore, this disclosure provides a block processing scheme for small-channel neuron data. When the number of channels in the neuron data is small, a suitable preferred alignment value P can be selected instead of using the baseline alignment value M of the block instruction, thereby reducing alignment requirements, more effectively utilizing the performance of the block instruction, and improving processing efficiency.

[0290] Similarly, weight data also suffers from the performance limitations mentioned above with neuron data. However, since the amount of weight data is usually small, the performance of weight data block processing has a very small impact on the overall computation performance, so the aforementioned optimization scheme is unnecessary. On the other hand, since the kw and kh dimensions of the weights are generally small, for example, kh*kw = 5*5, filling the ci direction with the wi direction will not add much. Forcibly aligning kw to Ws before weight block processing can be done using the above scheme, but this alignment itself already impacts performance, and the benefits may be insignificant. In addition, although other dimensions (such as hi, co, etc.) can be used to fill the weight data, the processing complexity will increase significantly, and both the preceding and following table arrangements will need to be modified accordingly. The workload complexity is not worth the efficiency improvement.

[0291] This disclosure also provides a data processing method for executing block instructions using the aforementioned data processing apparatus. Those skilled in the art will understand that the method steps for executing block instructions correspond to the various features of the computing apparatus described above in conjunction with the accompanying drawings; therefore, the features described above also apply to the method steps and will not be repeated here.

[0292] This disclosure also provides a chip that may include the data processing apparatus of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a circuit board that may include the aforementioned chip.

[0293] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server computing clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0294] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0295] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0296] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0297] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0298] The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A data processing apparatus, comprising a control circuit, a first storage circuit, and a second storage circuit, wherein: The first storage circuit is used to store input data before processing, wherein the input data includes input feature maps and convolution kernels; The second storage circuit is used to store the processed output data; and The control circuit is used for: The preferred alignment value is determined based on the size of the lowest storage dimension of the input data before processing; Configure block instructions according to the preferred alignment value; as well as The block instruction is executed to split the input data stored in the first storage circuit according to the first dimension storage order into split units and store them as output data in the second storage circuit. In the second storage circuit, each split unit is stored in the second dimension storage order, and the split units are stored in the third dimension storage order. The shape of the splitting unit is CHW=U Ci ×U H ×U W The preferred alignment value P is U Ci 2 n times, The control circuit is further configured to determine the preferred alignment value P based on the minimum storage dimension size ci of the input data according to the following rule: When 0 <ci≤U Ci At that time, P = U Ci , 2 n *U Ci <ci≤2 n+1 *U Ci , time, P=2 n+1 *U Ci , when At that time, P = M; The reference alignment value M is also known as the maximum number of operations performed by the hardware in a single operation.

2. The data processing apparatus according to claim 1, wherein the control circuit is further configured to: When the preferred alignment value P is less than the base alignment value M of the block instruction, the input data is regarded as an optimized shape hi*(wi / Ws)*(Ws*ci) from its original shape hi*wi*ci, where Ws=M / P, indicating that the data is split from the second lowest storage dimension W of the input data by Ws times and added to the lowest storage dimension C, where hi is the size of the highest storage dimension H of the input data, wi is the size of the second lowest storage dimension W, and ci is the size of the lowest storage dimension C.

3. The data processing apparatus according to claim 1, wherein the control circuit is further configured to configure block instructions as follows: When P≠U Ci At that time, according to the preferred alignment value P, a pre-alignment table is set for the block instruction. The pre-alignment table is used to rearrange the inner lowest dimension data of the input tensor of the block instruction according to the instructions of the pre-alignment table.

4. The data processing apparatus according to claim 3, wherein the control circuit is further configured to set the pre-matching table as follows: The inner lowest dimension data of the input tensor of the block instruction is transformed into a shape aligned with the splitting unit, and the input data of each splitting unit is arranged according to the storage order of the first dimension.

5. The data processing apparatus according to claim 1, wherein the control circuit is further configured to configure block instructions as follows: A post-allocation table is set for the block instruction, which is used to rearrange the inner lowest dimension data of the output tensor of the block instruction according to the instructions of the post-allocation table.

6. The data processing apparatus according to claim 5, wherein the control circuit is further configured to set the post-matching table as follows: The inner lowest dimension data of the output tensor, which is stored in the first dimension order, is converted to be stored in the second dimension order.

7. The data processing apparatus of claim 6, wherein the control circuit is further configured to configure block instructions as follows: In the block division instruction, the inner lowest dimension size in0 of the input tensor is set to ci*dwidth*Ws, the inner lowest dimension size in1 is set to max(1,4 / Ws), and the inner highest dimension size in2 is set to U. H dwidth is the bit width of the data to be divided into blocks; and Based on the size of each dimension of the optimized shape of the input data, set the size values ​​in3, in4 and in5 of the three outer dimensions of the input tensor in the block instruction, where the size values ​​of the three outer dimensions represent the number of inner data blocks defined by the three inner dimensions in the corresponding dimension.

8. The data processing apparatus of claim 7, wherein the control circuit is further configured to configure block instructions as follows: In the block instruction, the inner lowest dimension size on0 of the output tensor is set to M, the inner lowest dimension size on1 is set to in0 / T, the inner highest dimension size on2 is set to 1, and T is the block width of the block instruction, representing the amount of data in one atomic operation of the block instruction; and Based on the size of each dimension of the optimized shape of the input data, set the size values ​​on3, on4 and on5 of the three outer dimensions of the output tensor in the block division instruction, where the size values ​​of the three outer dimensions represent the number of inner data blocks defined by the three inner dimensions in the corresponding dimension.

9. The data processing apparatus of claim 8, wherein the control circuit is further configured to configure block instructions as follows: Based on the size of each of the six dimensions of the input tensor and the output tensor, and the size of each dimension of the optimized shape of the input data, the step size of adjacent data points in the storage space is set in the other five dimensions, excluding the lowest inner dimension.

10. The data processing apparatus of claim 9, wherein the control circuit is further configured to configure block instructions as follows: Set the block width T in the block instruction to 16B / max(1,4 / Ws).

11. The data processing apparatus according to claim 10, wherein when U Ci ×U H ×U W When M = 4B × 4 × 4 and M = 64B, the parameters of the block instruction are configured according to Table 1: Table 1 Where ci, hi, and wi represent the number of data items in the C, H, and W dimensions of the input data, respectively; dwidth represents the data bit width; B represents bytes; T represents the block bit width; is1 to is5 represent the five-dimensional step size of the input tensor; and os1 to os5 represent the five-dimensional step size of the output tensor.

12. The data processing apparatus according to claim 1, wherein the input data is neuron data comprising three dimensions: H, W, and C, wherein the first dimension is stored in the order HWC, the second dimension in the order CHW, and the third dimension in the order HWC, wherein C represents the input channel dimension, H represents the height dimension, and W represents the width dimension.

13. A chip comprising a data processing apparatus according to any one of claims 1-12.

14. A circuit board comprising the chip according to claim 13.

15. A data processing method for executing block instructions on input data using a data processing apparatus according to any one of claims 1-12.

Images