Data processing device, data processing method and related products

Abstract

The present disclosure discloses a data processing device, a data processing method using the data processing device to execute block instructions, and related products. The data processing device can be included in a combined processing device as a computing device, and the combined processing device can also include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete the computing operations specified by the user. The combined processing device can also include a storage device, which is respectively connected to the computing device and other processing devices, and is used to store data of the computing device and other processing devices. The scheme disclosed in the present disclosure realizes the data dimension conversion in the small convolution operation and improves the operation processing efficiency.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing, and more specifically, to a data processing device, a data processing method for executing block instructions on data using the data processing device, a chip, and a board. Background Art

[0002] At present, deep learning has become an important branch of machine learning and has greatly promoted the development of artificial intelligence (AI). The core technology of deep learning, deep neural network (DNN), has been widely used in many industries.

[0003] Neural network is one of the most critical technologies in artificial intelligence and deep learning, among which convolutional neural network (CNN) is the most important type of network. The most critical calculation in convolutional neural network is the convolution operation of the convolution layer (Conv layer). The function of the convolution layer is to extract features from the input data. Through multiple layers of convolution, complex features can be extracted to ensure that the network has sufficient expression and generalization capabilities. The neural network model contains a large number of convolution operations of various types, and the computational performance of convolution operations greatly affects the computational performance of the entire neural network model. When the neural network model is applied to different fields, such as speech recognition, machine translation, image processing, etc., the corresponding input feature maps and weights may have different dimensions. In order to fully utilize the hardware advantages of deep learning processors, it is necessary to optimize convolution operations of different scales and types to improve the computational performance of executing neural network models. Summary of the invention

[0004] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a data processing device in multiple aspects, which can make data of various dimensions adapt to the hardware of convolution operation by executing block instructions on data, thereby improving the computational efficiency of convolution operation. The convolution operation of the embodiment of the present disclosure can be an operation in various neural network models, which can be applied to various fields, such as image processing, speech processing, text processing, etc. These processes can include, but are not limited to, recognition and classification.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing device, including a control circuit, a first storage circuit, and a second storage circuit, wherein: the first storage circuit is used to store first data before processing; the second storage circuit is used to store second data after processing; and the control circuit is used to configure and execute a block instruction to convert the first data stored in the first storage circuit according to a first dimensional storage order into second data stored in the second storage circuit according to a second dimensional storage order, wherein the first data is three-dimensional data, and its three-dimensional shape is:

[0006] [1×1×co]

[0007] The second data is also three-dimensional data, and its three-dimensional shape is:

[0008]

[0009] Where co represents the output channel dimension of the data, group represents the number of groups in the grouping mode in the associated convolution operation, Indicates that each data in co is copied Second-rate.

[0010] In a second aspect, an embodiment of the present disclosure provides a chip, which includes the data processing device of the first aspect.

[0011] In a third aspect, the present disclosure provides a board comprising the chip of the second aspect.

[0012] In a fourth aspect, an embodiment of the present disclosure provides a data processing method for executing a block instruction on offset data using the data processing device of the first aspect.

[0013] Through the data processing device, chip, board and data processing method of executing block instructions by the data processing device as provided above, the scheme of the disclosed embodiment performs block processing on the bias data in various convolution splitting schemes to adapt to the processing capabilities of the hardware computing device, thereby making full use of the parallel processing capabilities of multiple slave processing circuits, and effectively improving the computational efficiency of the convolution operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0015] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0016] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0017] Figure 3a A schematic diagram showing the internal structure of a processor core of a single-core computing device according to an embodiment of the present disclosure;

[0018] Figure 3b A simplified schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0019] Figure 4 An example of an exemplary convolution operation principle to which the disclosed embodiments can be applied is shown;

[0020] Figure 5 A schematic structural block diagram of a computing device according to an embodiment of the present disclosure is shown;

[0021] Figure 6 An exemplary data storage sequence according to an embodiment of the present disclosure is shown;

[0022] Figure 7a-7c Several exemplary grouping modes according to embodiments of the present disclosure are shown;

[0023] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of the present disclosure is shown;

[0024] Fig. 9 A schematic diagram showing the splitting and storage of the Forward4 solution according to an embodiment of the present disclosure is shown;

[0025] Fig.10 A schematic diagram showing the division of output points of the operation circuit in the Forward4 solution according to an embodiment of the present disclosure is shown;

[0026] Fig.11 A schematic diagram showing a single operation in the Forward4 scheme according to an embodiment of the present disclosure is shown;

[0027] Fig.12 A schematic diagram of sliding convolution in the Forward4 scheme according to an embodiment of the present disclosure is shown;

[0028] Fig.13 A schematic diagram showing the output data format of the Forward4 solution according to an embodiment of the present disclosure;

[0029] Fig.14 The overall data handling process according to the embodiment of the present disclosure is shown;

[0030] Fig.15 A schematic conceptual diagram showing Trans Tiling according to an embodiment of the present disclosure;

[0031] Fig.16 A schematic diagram showing the front and rear matching table; and

[0032] Fig.17 A schematic diagram showing execution of a block instruction on offset data according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0034] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0035] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0036] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0037] Exemplary Hardware Environment

[0038] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1As shown, the board 10 includes a chip 101, which is a system-on-chip (SoC), or system on chip, integrated with one or more combined processing devices, which are artificial intelligence computing units that support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage, and powerful computing power.

[0039] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0040] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0041] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0042] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0043] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0044] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0045] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is usually 16G or larger in size, and is used to store data of the computing device 201 and / or the processing device 203 .

[0046] Figure 3a The internal structure diagram of the processing core when the computing device 201 is a single-core device is shown. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three modules: a control module 31, a computing module 32, and a storage module 33.

[0047] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the deep learning task, and includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to obtain instructions from the processing device 203, and the instruction decode unit 312 decodes the obtained instructions and sends the decoding results to the operation module 32 and the storage module 33 as control information.

[0048] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, namely matrix multiplication and convolution.

[0049] The storage module 33 is used to store or transfer relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access module (DMA) 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; WRAM 332 is used to store the convolution kernel of the deep learning network, that is, the weight; DMA 333 is connected to DRAM 204 through bus 34, and is responsible for data transfer between the computing device 301 and DRAM 204.

[0050] Figure 3b The figure shows a simplified schematic diagram of the internal structure of the computing device 201 as a multi-core. The multi-core computing device can be abstracted using a hierarchical hardware model. As shown in the figure, the multi-core computing device can be abstracted into four levels, namely, the card level (Card) 350, the chip level (Chip) 360, the processor cluster level (Cluster) 370 and the processor core level (Core) 380. The disclosed embodiment mainly involves the data transmission and computing unit part of the storage unit, so the drawings and descriptions briefly show and introduce the relevant computing structure, and omit other parts.

[0051] At the board level, each board contains local DDR storage, and each processor chip serves as a computing and control unit.

[0052] At the chip level, each processor chip contains multiple multiprocessors as computing units.

[0053] At the computing cluster level, each multiprocessor includes multiple accelerator cores as control and computing units, and a shared memory SRAM as a storage unit.

[0054] At the processor core level, each accelerator core contains local storage and a local processing unit array. NFU refers to the Neuron Function Unit, which is used for convolution calculations.

[0055] In this multi-core computing device, the storage model includes the global memory of the board, SRAM (shared memory) on the Cluster, NRAM, WRAM and registers on the Core, etc. In order to obtain better performance, the data movement between the storage levels below the Card and the balance between memory access / computation can be explicitly controlled. SRAM is contained in the storage processing unit MPU (Memory Process Unit Core, referred to as MPU, or Mem Core). Core refers to the intelligent processing core (Intelligent Process Unit Core, referred to as IPU Core or Core) in the multi-core computing device. 1 IPU Core contains NRAM, WRAM, NFU, etc. Cluster refers to a processor cluster or a computing cluster. Usually a multi-core computing device contains several Clusters, and a Cluster contains 1 Mem Core + N IPU Cores.

[0056] Example convolution operation types

[0057] The convolution layer in the neural network model can perform convolution operations, and extract features by applying convolution kernels (also called filters, weights, etc.) to the input feature map (also called input data, neurons, or input neurons) for convolution processing. The convolution layer can contain multiple convolution kernels, and each element of the convolution kernel corresponds to a weight coefficient and a bias. The disclosed embodiments can be applied to data splitting of various convolution operations.

[0058] In conventional 3D convolution operations, assuming that the input feature map tensor shape in the convolution layer is represented by X[N Hi Wi Ci], the tensor shape of the convolution kernel is represented by K[Co Kh Kw Ci], and the output result is Y[N HoWo Co], then the simplified mathematical calculation formula of the convolution operation can be expressed as follows:

[0059] Y in,jc,jh,jw =∑ 0≤ic≤ci,0≤ih≤kh,0≤iw≤kw X in,ic,jh×sh+ih,jw×sw+iw ×K jc,ic,ih,iw (1)

[0060] In the above formula, X is the input data, Y is the output data, K is the convolution kernel, Kh and Kw are the length and width of K, sh and sw are the strides in the length and width directions, and the formula ignores bias, padding and dilation, and assumes that the input data X has been padded and the convolution kernel has been dilated. The formula ignores the N dimension and the C dimension. The forward calculation of the neural network model in the N dimension is independent and fully connected in the C dimension. When the convolution kernel is working, it will scan the input features at a certain step length, perform matrix element multiplication on the input features in the convolution window, and superimpose the deviation.

[0061] Figure 4 An exemplary conventional 3D convolution operation principle example to which the disclosed embodiments can be applied is shown.

[0062] The figure exemplarily shows four-dimensional input data X of size [N Hi Wi Ci], which can be represented as N three-dimensional rectangles of size Hi×Wi×Ci 410. The figure also exemplarily shows a four-dimensional convolution kernel K of size [Co Kh Kw Ci], which can be represented as Co three-dimensional convolution kernels of size Kh×Kw×Ci 420. The convolution result of the input data X and the convolution kernel K obtains the output data Y, which is four-dimensional data of size [N Ho Wo Co], which can be represented as N three-dimensional rectangles of size Ho×Wo×Co 430.

[0063] The figure also specifically shows an example of a convolution operation, in which the input data is an input feature map 440 of size 6×6×3, omitting the N dimension; the convolution kernel is a stereo convolution kernel 450 of size 3×3×3, targeting a single Co; and the output data is a 4×4 output feature map 460. The specific operation process is as follows:

[0064] The convolution kernel 450 scans the input feature map 440 at a certain step length, performs matrix element multiplication and summation on the input features in the convolution window 470, and superimposes the bias. That is, the value at each position in the output feature map 460 is obtained by performing a two-dimensional convolution operation on the corresponding block of each input feature map and the corresponding convolution kernel, and then adding them together. For example, the figure shows that the value at the (0,0) position on the output feature map 460 (that is, the convolution output point) is obtained by performing a two-dimensional convolution operation on the convolution window 470 framed by the black cube in the input feature map and the three-dimensional convolution kernel 450 to obtain three values, and then adding them together to obtain the final value.

[0065] In order to obtain outputs at other positions, the position of the convolution kernel 450 can be moved on the input feature map 440, that is, the convolution window of the convolution output point can be moved. In the example in the figure, the convolution step size (Sx, Sy) is (1,1). When the convolution operation is performed after moving one grid to the right horizontally (width direction) or downward vertically (height direction), the value of the position (0,1) or (1,0) on the output feature map 460 can be obtained respectively.

[0066] From the above description, we can see that in a convolutional layer of a neural network, there are N groups of input feature maps, each group contains Hi×Wi×Ci information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The convolutional layer has Ci×Co convolution kernels of size Kh×Kw, where Ci is the number of input channels, Co is the number of output feature maps (or the number of output channels), and Kh and Kw are the height and width of the convolution kernel, respectively. The output feature map contains Ho×Wo×Co information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. In addition, in the convolution operation, the convolution step size (Sx, Sy) is also involved, and the size of the convolution step size will affect the size of the output feature map.

[0067] In this article, input feature map, input data, neuron or input neuron can be used interchangeably; convolution kernel, filter or weight can be used interchangeably. In addition, H (height) and Y dimension can be used interchangeably, and W (width) and X dimension can be used interchangeably. Accordingly, the H dimension of the input feature map can be expressed as Hi or Yi, the H dimension of the output feature map can be expressed as Ho or Yo, and the W dimension is expressed similarly. In the disclosed embodiment, each convolution output point has a corresponding convolution window, and the shape of the convolution window is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and the weight within its convolution window.

[0068] Exemplary computing device / data processing device

[0069] In the disclosed embodiment, a master-slave computing device may be used to implement the above convolution operation. Furthermore, different data paths may be configured for the input feature map and the convolution kernel to improve memory access efficiency.

[0070] Figure 5 FIG. 5 is a schematic structural block diagram of a computing device 500 according to an embodiment of the present disclosure. It can be understood that the structure can be regarded as a refinement of the internal structure of the computing module of a single processing core in FIG. 3 , or can be regarded as a functional division block diagram based on the computing modules of multiple processing cores shown in FIG. 3 . Figure 5As shown, the computing device 500 of the disclosed embodiment can be configured to perform various types of convolution operations, which may include a master processing circuit (MA) 510 and multiple slave processing circuits (SL) 520, and 16 slave processing circuits SL0 to SL15 are shown in the figure. Those skilled in the art will appreciate that the number of slave processing circuits may be more or less, depending on the specific hardware configuration, and the disclosed embodiment is not limited in this respect.

[0071] The master processing circuit and the slave processing circuit and the multiple slave processing circuits can communicate with each other through various connections. In different application scenarios, the connection mode between the multiple slave processing circuits can be a hard connection mode arranged by hard wires, or a logical connection mode configured according to, for example, microinstructions, to form a topological structure of multiple slave processing circuit arrays. The disclosed embodiments are not limited in this regard. The master processing circuit and the slave processing circuit can cooperate with each other to achieve parallel computing processing.

[0072] In order to support the computing functions, the master processing circuit and the slave processing circuit may include various computing circuits, such as vector operation units and matrix operation units. The vector operation unit is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit is responsible for the core calculations of deep learning algorithms, such as matrix multiplication and convolution.

[0073] The slave processing circuit may be used, for example, to perform intermediate operations on corresponding data in parallel according to the operation instruction to obtain multiple intermediate results, and transmit the multiple intermediate results back to the master processing circuit.

[0074] By configuring the computing device 500 to be a master-slave structure (e.g., a one-master-and-multiple-slave structure, or a multiple-master-and-multiple-slave structure, and the present disclosure has no limitations in this regard), for forward operation calculation instructions, the data can be split according to the calculation instructions, so that the parts with larger calculation amount can be calculated in parallel through multiple slave processing circuits to increase the calculation speed, save calculation time, and thereby reduce power consumption.

[0075] In some embodiments of the present disclosure, by utilizing different data paths to transmit input feature maps and weights, multiple multiplexing methods of input feature maps and weights can be supported, thereby reducing the amount of data access during calculations and improving processing efficiency.

[0076] Specifically, the computing device 500 may further include a first storage device 530 and a second storage device 540 for respectively storing data transmitted via different data channels.

[0077] The first storage circuit 530 can be used to store multicast data, that is, the data in the first storage circuit will be transmitted to multiple slave processing circuits through the broadcast bus, and these slave processing circuits receive the same data. It can be understood that broadcast and multicast can be realized through the broadcast bus. Multicast refers to a communication method that transmits a copy of data to multiple slave processing circuits; while broadcast is a communication method that transmits a copy of data to all slave processing circuits, which is a special case of multicast. Since multicast and broadcast both correspond to one-to-many transmission methods, the two are not specifically distinguished in this article. Broadcast and multicast can be collectively referred to as multicast, and those skilled in the art can clarify their meanings based on the context.

[0078] The second storage circuit 540 may be used to store distribution data, that is, the data in the second storage circuit will be transmitted to different slave processing circuits respectively, and each slave processing circuit receives different data.

[0079] By providing the first storage circuit and the second storage circuit respectively, it is possible to support the transmission of the data to be operated in different transmission modes, thereby reducing the data memory access amount by multiplexing the multicast data among multiple slave processing circuits.

[0080] In some embodiments, the master processing circuit may determine one of the input feature map and the convolution kernel as multicast data and store it in a first storage circuit, so as to transmit the data to the scheduled multiple slave processing circuits by broadcasting during the operation. Correspondingly, the master processing circuit may determine the other of the input feature map and the convolution kernel as distributed data and store it in a second storage circuit. These distributed data may be distributed to the corresponding slave processing circuits before the operation.

[0081] Figure 5 A schematic diagram of the internal structure of the slave processing circuit SL according to the embodiment of the present disclosure is also shown. As shown in the figure, each slave processing circuit 520 may include multiple operation circuits CU 521, a first buffer circuit 522 and a second buffer circuit 523. Four operation circuits CU0-CU3 are shown in the figure. It can be understood by those skilled in the art that the number of operation circuits can be more or less, depending on the specific hardware configuration, and the embodiment of the present disclosure is not limited in this respect.

[0082] In some embodiments, the first buffer circuit 522 can be used to cache the weights or input feature maps assigned to the slave processing circuit. Correspondingly, the second buffer circuit 523 can be used to cache the input feature maps or weights assigned to the slave processing circuit. Both buffer circuits are used to select data involved in the operation. The data of the first buffer circuit 522 can be multiple data rows from, for example, the first storage circuit 530 or the second storage circuit 540, and correspondingly, the data of the second buffer circuit 523 can be multiple data rows from, for example, the second storage circuit 540 or the first storage circuit 530. Depending on the specific multiplexing method, these data rows can be distributed to the corresponding operation circuit CU 521 or broadcast to all CU 521 in the slave processing circuit 520 during the operation.

[0083] Each operation circuit CU 521 is used to perform a bitwise multiplication and accumulation operation on the data row selected from the first buffer circuit and the data row selected from the second buffer circuit respectively during each calculation.

[0084] By providing a first buffer circuit and a second buffer circuit respectively, it is possible to support the transmission of data to be operated in different transmission modes, thereby reducing the amount of data memory access by multiplexing data as much as possible between multiple operation circuits in a single slave processing circuit.

[0085] The slave processing circuit 520 may further include a third buffer circuit 524 for buffering the operation results of each operation circuit CU 521 .

[0086] Understandably, although Figure 5 The various processing circuits and storage circuits are shown as separate modules, but according to different configurations, the storage circuits and processing circuits can also be combined into one module. For example, the first storage circuit 530 can be combined with the main processing circuit 510, and the second storage circuit 540 can be shared by multiple slave processing circuits 520, and an independent storage area is allocated to each slave processing circuit to speed up access. The disclosed embodiments are not limited in this regard. In addition, in the computing device, the main processing circuit and the slave processing circuit can belong to different modules of the same processor or chip, or to different processors, and the disclosure is not limited in this regard.

[0087] Exemplary Data Splitting and Storage

[0088] In the disclosed embodiment, the dimensions of the multidimensional data involved are characterized as (N, H, W, C) or (Co, H, W, Ci), which represents the storage order of the data in the memory. It can be understood that although the multidimensional data has multiple dimensions, because the layout of the memory is always one-dimensional, there is a corresponding relationship between the multidimensional data and the storage order on the memory. Multidimensional data is usually allocated in a continuous storage space, that is, the multidimensional data can be expanded in one dimension and stored in sequence on the memory. For example, in the disclosed embodiment, the initial input feature map can be stored sequentially in a low-dimensional (here C / Ci is the lowest dimension) priority manner; and in order to optimize the convolution operation, the storage order of the input feature map can be adjusted during the operation, as will be described in detail later. Adjacent dimensions refer to dimensions that are adjacent to each other in the dimensional information representation of multidimensional data. For example, W and Ci are adjacent, and adjacent dimensions can also be called continuous dimensions.

[0089] In intelligent processors, due to the need for computing power and considerations of area and power consumption, the main hardware operation unit is the vector multiplication and addition operator. Implementing support for various convolution algorithms in hardware design is essentially to maximize the extraction of multiplication and addition operations in the algorithm, and to efficiently exchange the input and output data of the multiplication and addition operations between the on-chip RAM (such as NRAM, WRAM, etc. in Figure 3) and the operator through the data path.

[0090] Hardware stores data in rows (cache rows) and read, write and compute operations are most efficient when the entire row is aligned. Therefore, in order to fully utilize bandwidth and adapt to the memory access requirements of the operator array, it is usually necessary to vectorize and align the data. The design of artificial intelligence chips usually takes the Ci dimension as the lowest dimension, that is, the NHWC arrangement order mentioned above, and the data on the Ci dimension is continuous. Therefore, vectorization alignment requires that the size of the Ci dimension be aligned to a specified value, such as the alignment value M, so that the access number is performed in units of the alignment value M, which can also be called the maximum single hardware operation. Based on different hardware designs, M can have different values, such as 64bit, 128bit, 256bit, 512bit, etc. Usually, the input port size of the operator array is also related to M. For example, when the input data bit width is symmetrical, the input port size of the operator array is usually twice that of M, that is, the input feature map data and weight data of the alignment value M scale are processed at one time. When the Ci dimension of the input feature map is large, it is easier to meet the above alignment requirements.

[0091] When the Ci dimension of the input feature map is small, for example, smaller than the size of a cache line, the Ci dimension needs to be padded to a row of data (for example, 512 bits), that is, filled with invalid data 0. This padding will cause a lot of redundant calculations, resulting in a waste of resources and reducing the efficiency of the operation.

[0092] In the disclosed embodiment, a convolution operation scheme is proposed, which can determine the corresponding convolution splitting scheme according to the size of the lowest storage dimension (e.g., Ci) of the input feature map, wherein the convolution splitting scheme at least indicates the shape of the splitting unit of the data to be operated. The amount of data contained in a splitting unit does not exceed the maximum single operation amount of the hardware.

[0093] In some embodiments, the amount of data contained in a split unit can be set to the hardware's one-time processing alignment value M, so that calculations are performed in units of split units, which can give full play to the computing power of the hardware and avoid or reduce invalid calculations.

[0094] In the exemplary description of the present disclosure, it is assumed that M = 512bit = 64Byte, the data type can be Int8, Int16, Float16 or Float32, and the input feature map is consistent with the data type of the convolution kernel. Since the data type requires at least 1 byte of width and the minimum unit of operation processing is one data, various calculations are performed in bytes in the following examples, such as M = 64B, Ci = 28B, etc., where units are sometimes omitted for simplicity.

[0095] When the data volume of the split unit is equal to M, the data block shape of each split unit is blockC×blockY×blockX, which may exist in many situations, and Table 1 lists several of them:

[0096]

[0097] Table 1. Data block shape

[0098] As can be seen from Table 1, some data block shapes have equal X and Y dimensions (as shown in the dark rows), and this shape can simplify subsequent operations. Therefore, in the disclosed embodiment, this data block shape can be preferably used to split the data to be operated.

[0099] For the sake of simplicity, the 64B×1×1 splitting scheme is called Forward64, the 16B×2×2 splitting scheme is called Forward16, the 4B×4×4 splitting scheme is called Forward4, the 4B×4×4 splitting scheme applied to the depth convolution operation is called Forward1, the 4B×4×4 splitting scheme applied to the reverse depth convolution operation is called Update1, and the 4B×4×4 splitting scheme applied to the cross multiplication convolution operation is called Update4. Except for Forward64, these splitting schemes are suitable for scenarios where the channel C is relatively small in the convolution calculation, so they can also be collectively referred to as small convolution. In these small convolution splitting schemes, a split unit includes data of the lowest storage dimension and at least one other storage dimension, and the total amount of data of a split unit does not exceed the maximum single operation amount of the hardware.

[0100] Different convolution splitting schemes can be applied to different computing scenarios, thereby achieving different degrees of performance optimization.

[0101] After the splitting scheme is determined, the input feature map and convolution kernel can be split into multiple corresponding splitting units according to the determined convolution splitting scheme and the dimensional storage order can be converted so that the data in a splitting unit can be stored continuously as a data row, thereby facilitating subsequent reading and processing in units of splitting units (data rows).

[0102] In some embodiments, three-dimensional or four-dimensional neuron or weight data are all divided into data blocks of size blockC×blockY×blockX (Uc×Uy×Ux), and each data block is continuously stored in a row, for example, M=64B, so that when a row of data is read, the data of a data block is actually taken out.

[0103] Specifically, one or more split units can be read in a first reading order from the data to be calculated stored in the first dimensional storage order, with the split units as units, and the read split units can be stored in a corresponding storage circuit, wherein the data in each split unit is stored in the second dimensional storage order, and the split units are stored in the third dimensional storage order.

[0104] Figure 6 An exemplary data storage order according to an embodiment of the present disclosure is shown.

[0105] As shown in the figure, 610 represents the storage method of the four-dimensional tensor to be calculated, which includes N 3-dimensional sub-tensors, and N is in the highest dimension, that is, the first dimension of the four-dimensional tensor. The storage order is NHWC. Note that H and Y, W and X are used interchangeably in this article. Each sub-tensor is divided into smaller data blocks or split units, and the number of data blocks in each dimension is C / Y / X respectively.

[0106] The middle figure 620 shows the storage method of each sub-tensor, and each data block is stored as a continuous 64Byte, that is, a row. When the order of reading the data blocks is different, the order between the rows will change accordingly. In the example in the figure, the data blocks are read in the direction of C first, then X, and finally Y, that is, the first reading order is YXC, then the rows are stored in the order of Y×X×C, that is, the third dimension storage order is YXC or HWC. In this example, the third dimension storage order is the same as the first dimension storage order. It can be understood that other reading orders can also be used, which will cause the third dimension storage order to be different from the first dimension storage order, and they will not be listed here one by one.

[0107] The diagram 630 on the right represents the order within each row, that is, the data order within each data block, and its shape is blockC×blockY×blockX. In this case, the storage order of the second dimension is CYX or CHW.

[0108] Example Grouping Operation

[0109] Small convolution uses block form. Its advantage over traditional convolution is that the alignment in the Ci direction only needs to satisfy the alignment of the block in the Ci direction. In this small channel scenario, the weights (co×Kh×kw×ci) are generally small, Kh and Kw are usually in the single digit, and co and ci are similar. Figure 5 In the described computing device / data processing device, the storage space of the second storage circuit (e.g., WRAM 332 in FIG. 3 ) is usually larger than that of the first storage circuit (e.g., NRAM 331 in FIG. 3 ). Therefore, in order to make full use of the on-chip computing space, in most small convolution schemes, such as Forward4, Forward1, etc., a scheme of exchanging the storage locations of neurons and weights with those of normal convolution is adopted, that is, neurons are stored in the second storage circuit WRAM, and weights are stored in the first storage circuit NRAM.

[0110] The calculation of convolution is that each input feature map needs to be multiplied and added with each Co convolution kernel, so as to output Co output feature maps. However, the on-chip space is not necessarily able to store all sizes of convolution kernels and input feature maps at the same time. Therefore, for the hardware, there is a series of operations of repeatedly loading input feature data or weight data. How to balance the repeated loading of input feature data or weight data will have a certain impact on the efficiency of the calculation. In actual calculations, in order to reduce frequent off-chip memory accesses, there is a problem of splitting strategies for neurons and weights. In some embodiments, different splitting methods can be adopted according to the scale characteristics of the data involved in the operation.

[0111] According to the convolution operation principle described above, the operation results on the Co dimension (C dimension for depth convolution) do not need to be accumulated, so the operations on different Co can be performed relatively independently on different operation circuits. In small convolution scenarios, usually the size of the Co dimension of the output channel of the convolution kernel in a single round of operation does not exceed the number of scheduled slave processing circuits, so the operation of a single Co needs to be completed by one or more slave processing circuits. More generally, even when the Co dimension is large, it can be achieved by splitting it into multiple rounds of operations, in which the size of Co processed in each round of operations does not exceed the number of scheduled slave processing circuits. Therefore, in one example, the number of operation rounds required to complete the convolution operation and the number of Co processed in each round of operations or the corresponding grouping mode can be determined first based on the Co dimension size of the output channel of the convolution kernel and the number of schedulable slave processing circuits Ns.

[0112] Regardless of the allocation method, in a single round of operation, Co may have two allocation situations: multiple slave processing circuits process one Co value, or a single slave processing circuit processes one or more Co values. Specifically, in a single operation round that processes Nco output channels, each Rs SL constitutes a slave processing circuit group SLB, which processes the convolution kernel corresponding to the same output Co value, Rs = [Ns / Nco], that is, the same convolution kernel is reused on Rs SLs in the same SLB, and Rs represents the number of times the convolution kernel is reused between slave processing circuits. Correspondingly, the input feature map can be reused between each slave processing circuit group SLB, Rn = [Ns / Rs], which represents the number of times the input feature map is reused between slave processing circuits.

[0113] Optionally or additionally, when each slave processing circuit processes convolution kernels corresponding to rn Co values, rn = [Nco / Ns], the input feature map processed by each slave processing circuit can be reused for rn convolution kernels, rn represents the number of times the input feature map is reused in a single slave processing circuit. Factors such as hardware buffer space limitations (e.g. Figure 5The size of the first buffer circuit and the second buffer circuit in the buffer circuit are used to determine the maximum number of convolution kernel multiplexing times rs and the maximum number of input feature map multiplexing times rn that can be applied within a single slave processing circuit.

[0114] Taking into account the cache size limitations and reuse benefits in the hardware circuits, in some embodiments of the present disclosure, the case where a slave processing circuit processes multiple Co values ​​in a single round of operations is not temporarily considered, but only the case where one or more slave processing circuits only process one Co value in a single round of operations is considered.

[0115] Different grouping modes can be used according to the number of slave processing circuits SL that process the same Co value in a single round of operation. It can be understood that it is preferred to evenly distribute the available slave processing circuits SL to balance the computing power, for example, every 2 SLs are grouped together, so that 16 SLs can process 8 Co values ​​at the same time; or every 4 SLs are grouped together, so that 16 SLs can process 4 Co values ​​at the same time; and so on. Figure 5 In the computing device described in the embodiment, the second storage circuit WRAM has 16 storage areas, which are respectively allocated to the 16 slave processing circuits SL. Furthermore, every 4 blocks can be combined into a storage block, which is allocated to the corresponding slave processing circuit group SLB. Therefore, in some embodiments, for Figure 5 The computing device shown includes Ns=16 SLs, and the following grouping modes can be selected: Group 1 mode, Group 4 mode and Group 16 mode. Those skilled in the art can understand that different grouping modes can be used according to different values ​​of Ns, and each grouping mode can refer to the above three representative grouping modes given in this article for corresponding processing.

[0116] In some embodiments, the above grouping mode can be uniformly expressed as GroupN, which means that all slave processing circuits SL scheduled in the current round of operation are divided into N groups, each slave processing circuit group SLB processes the same Co value, and different slave processing circuit groups SLB process different Co values. For a total of 16 SLs that can be scheduled, N can be 1, 4, or 16, corresponding to Group1, Group4, and Group16 above, respectively.

[0117] Figure 7a-7c Several exemplary grouping modes according to embodiments of the present disclosure are shown. Figure 7a Group1 mode is shown. Figure 7b Group16 mode is shown. Figure 7c A Group 4 mode is shown.

[0118] like Figure 7aAs shown, Group1 mode means that all 16 schedulable SLs belong to one group and process one Co value together. For example, SL0 to SL15 belong to group G0. Therefore, the operation for the one output channel is distributed on 16 SLs. In this mode, the convolution kernel 720 of the output channel can be preferentially transmitted to each SL in a broadcast manner, and the input feature map 710 is split and distributed to each SL, thereby improving memory access efficiency.

[0119] In one embodiment, the convolution kernel can be stored in Figure 5 The input feature map is stored in the first storage circuit 530 for transmission via the broadcast channel. The input feature map can be divided according to the XY direction of the output feature map and stored in the second storage circuit 540 to be allocated to different SLs. Thus, all SLs jointly calculate an output feature map of Co. The division and storage of the input feature map will be described in detail later in conjunction with the accompanying drawings.

[0120] like Figure 7b As shown, the Group16 mode means that all 16 schedulable SLs are divided into 16 groups, that is, one SL in each group, and each SL processes a different Co value. For example, SL0 belongs to group G0, SL1 belongs to group G1, and so on, until SL15 belongs to group G15. In this mode, the same input feature map 730 can be reused among the 16 SLs, so it can be given priority to transmit the input feature map 730 to each SL in a broadcast manner, and the convolution kernels 740 corresponding to different Co are distributed to the corresponding SLs.

[0121] In one embodiment, the input feature map can be copied 16 times and stored in 16 storage areas allocated to 16 slave processing circuits on the second storage circuit. The convolution kernel is divided according to Co, one SL corresponds to one Co, 16 Cos are processed at a time, stored in the first storage circuit, and distributed to different SLs in a unicast manner. As a result, all SLs calculate output feature maps of different Cos for the same input feature map.

[0122] like Figure 7c As shown, Group4 mode means that all 16 schedulable SLs are divided into 4 groups, each group processing one Co value. The number of SLs included in each SL group (abbreviated as SLB) is equal to Rs=Ns / 4=4. For example, SL0~SL3 belong to group G0, SL4~SL7 belong to group G1, SL8~SL11 belong to group G2, and SL12~SL15 belong to group G3. This mode is between Group1 and Group16, so either the convolution kernel or the input feature map can be determined as multicast data, and the other can be determined as distributed data.

[0123] In one embodiment, the convolution kernels can be divided into 4 groups according to Co and stored in Figure 5 The input feature map can be divided into four parts according to the XY direction of the output feature map and copied into four parts, which are stored in the second storage circuit 540 to be distributed to the four SLBs. Each SLB obtains the same input feature map, which is then distributed to the four SLs in the SLB according to the four divided parts. Thus, all SLs in each SLB jointly calculate an output feature map of Co, and the four SLBs process a different Co respectively.

[0124] like Figure 7c As shown in the figure, the convolution kernel is divided into 4 groups, and each group is divided into each group according to Co with an interval of 1. For example, when Co = 12, the 4 groups of Co are {0, 4, 8}, {1, 5, 9}, {2, 6, 10} and {3, 7, 11}. Each time, a Co of each group is sent. For example, the first time Co = 0 to 3 is sent, one Co corresponds to one SLB, and the four SLs in one SLB share the same weight; the second time Co = 4 to 7 is sent, and so on. Therefore, after each round of operation is completed, the Co dimension of the operation result output by each SLB is continuous.

[0125] When the small convolution splitting operation scheme such as Forward4 is adopted, in order to support the above three modes at the same time, the neurons can be uniformly stored in the second storage circuit WRAM and the weights can be stored in the first storage circuit NRAM.

[0126] Example split of input feature map

[0127] From the previous description, it can be seen that when multiple SLs jointly process a Co value, the input feature map needs to be split between these multiple SLs. For example, the Group1 grouping mode needs to split the input feature map into 16 parts, while the Group4 grouping mode needs to split the input feature map into 4 parts.

[0128] In order to ensure that the split input feature map can share the convolution kernel, it can be divided according to the Ho / Wo direction of the output feature map, so as to map back to the division of the input feature map. In some embodiments, the input feature map can be divided between the Rs slave processing circuits SL included in each slave processing circuit group as follows: according to the size of the corresponding output feature map, the output feature map is evenly divided into Rs output feature blocks of the same shape in the XY dimension (that is, the Ho / Wo dimension); and according to the input feature map area required to calculate each output feature block, the input feature map is divided into Rs input feature blocks in the XY dimension (that is, the Hi / Wi dimension) to be allocated to the Rs slave processing circuits. It can be understood that depending on the convolution kernel size and the convolution step size, the input feature maps corresponding to adjacent output points on the output feature map may overlap.

[0129] Figure 8 An exemplary splitting diagram of an input feature map according to an embodiment of the present disclosure is shown. In this example, the input feature map is divided into 16 parts and distributed on 16 SLs, corresponding to the Group 1 mode.

[0130] In the figure, 810 represents the output feature map of a single Co, which is divided into 16 output feature blocks of the same shape in a 4×4 manner in the XY direction, and respectively allocated to SL0 to SL15. Then, these 16 output feature blocks can be mapped to the input feature map 820 to obtain the 16 input feature map regions required for calculating the 16 output feature blocks respectively, which also divides the input feature map in the XY direction. These 16 input feature map regions can be correspondingly allocated to the 16 slave processing circuits SL.

[0131] According to the above description, the input feature map will be split in units of splitting units according to the determined convolution splitting scheme. Therefore, the block division of the input feature map in the above embodiment should make each divided input feature map block a multiple of the XY dimension of the splitting unit in the XY direction, that is, it can be aligned according to the splitting unit in the XY direction. For example, when a 4×4×4 convolution splitting scheme is selected, each input feature map block is aligned as 4×4; and when a 16×2×2 convolution splitting scheme is selected, each input feature map block is aligned as 2×2.

[0132] If the output feature map is not aligned according to the splitting unit (for example, 4×4 or 2×2), it is necessary to pad the input feature map accordingly (for example, fill it with 0) so that the output XY actually calculated is aligned according to the splitting unit (for example, 4×4 or 2×2) and the input XY is also aligned according to the splitting unit (for example, 4×4 or 2×2).

[0133] Those skilled in the art will appreciate that the output feature map may also be split in the XY direction according to other rules, for example, split into 16 output feature blocks of the same shape in a 1×16 manner, and respectively assigned to SL0 to SL15. The disclosed embodiments are not limited in this regard. In addition, it can be understood that although the foregoing description is made in conjunction with the split between the slave processing circuits, this splitting method may also be applied to splitting in other scenarios, such as the splitting between the computing circuits CU within a single slave processing circuit SL, and the disclosed embodiments are not limited in this regard.

[0134] Example convolution operation process within a single slave processing circuit

[0135] After the data to be operated is split and stored accordingly, multiple slave processing circuits can be scheduled to perform convolution operations on the corresponding data rows of the input feature map and the convolution kernel. Then, according to the convolution splitting scheme, the operation results returned by the multiple slave processing circuits can be spliced ​​to obtain the output feature map of the convolution operation of the input feature map and the convolution kernel. Specifically, multiple operation circuits CU and each buffer circuit (see Figure 5 ) to perform the specific convolution operation process. Depending on the size of the buffer circuit inside the processing circuit and the computing power limit of the operation circuit, multiple operation cycles are usually required to complete the required operation in each round of operation.

[0136] As can be seen from the previous description, in the conventional 3D convolution operation scenario, all the operation circuits in a single slave processing circuit calculate an output feature map or a partial output feature map corresponding to the same output channel Co. Depending on the buffer space size of the first buffer circuit and the second buffer circuit in the slave processing circuit SL, and the processing capacity of the operation circuit CU (such as internal registers, etc.), the slave processing circuit may not be able to calculate the output feature map assigned to it at one time. Therefore, the output feature blocks can be divided into units based on the single operation capacity of the operation circuit (for example, a single calculation of Nop output points or partial sums), and each output feature block corresponds to all N schedulable blocks in a single SL. CU The single operation capability of the operation circuit (N CU ×Nop output points). For example, Figure 5 For example, each SL includes 4 CUs, assuming that each CU can calculate Nop=4 output points or partial sums of output points at a time, a single SL can calculate 4×4=16 output points (or partial sums) at a time. Therefore, the output feature map can be divided into output feature blocks according to the alignment of 16 output points in the XoYo dimension, and each output feature block can be calculated one by one. It can be understood that these 16 output points can be in the form of 4×4 or 1×16, and the disclosed embodiment has no limitation in this regard.

[0137] When calculating the output feature blocks of each partition, we can further CU The output points of the output feature block are divided between the operation circuits to determine the processing objects of each operation circuit. Then, according to the division of the output points, the split unit is used as a sliding window to select N from the first buffer circuit. CU The input feature data rows are distributed to N CU An operation circuit selects the corresponding weight data from the second buffer circuit and broadcasts it to N CUA computing circuit is provided to realize parallel calculation of output points corresponding to multiple sliding windows by reusing weight data. Nk sliding selections are performed, where Nk is determined according to the smaller value of the size of the convolution kernel in the X and Y dimensions and the maximum convolution kernel size supported by the processing circuit in a single operation in the current convolution split mode.

[0138] In some embodiments, when performing a conventional three-dimensional convolution operation, the corresponding weight data may be selected as follows: 1 / Nop weight rows are selected from the second buffer circuit in a sliding manner corresponding to that in the first buffer circuit, and Nop-1 copies are copied to expand it into an extended weight row, which is broadcasted to N weight rows in the slave processing circuit. CU An operational circuit.

[0139] At this time, each operation circuit can, during each sliding selection calculation, perform bitwise multiplication and accumulation in units of 1 / Nop data rows for an input feature row from the first buffer circuit and an extended weight data row from the second buffer circuit to obtain Nop partial sums; and accumulate the Nk×Nop partial sums obtained from Nk sliding selection calculations according to the corresponding convolution output points to obtain and output Nop operation results.

[0140] When the slave processing circuit outputs the output points of the operation circuit therein, the output points calculated by the multiple operation circuits therein can be output in a specific order according to the division method of the output points, so that the output points outputted continuously are continuous in the X and / or Y dimensions, which is convenient for subsequent processing. In some embodiments, the main processing circuit can further store the operation results returned from each slave processing circuit in a fourth dimensional storage order. Depending on the situation, the main processing circuit can also convert the operation results into a desired dimensional storage order for storage.

[0141] There are many ways to divide the output points between the operation circuits, and the sliding selection convolution process and the output order of the output points are also different accordingly.

[0142] The following is a detailed description of the entire data splitting, storage, convolution sliding and calculation output process in combination with the Forward4 solution.

[0143] The shape description of the input neurons and weights of the Forward4 scheme

[0144] In Forward4, the shape of the split unit block is 4B × 4 × 4. The shape of the block is slightly different depending on the data type. Table 2 shows the block shape of Forward4 under different data types.

[0145]

[0146] Table 2. Data block shapes of Forward4 under different data types

[0147] Fig. 9 The figure shows a schematic diagram of splitting and storing the Forward4 solution according to an embodiment of the present disclosure. For simplicity, the example in the figure assumes that the data type is Int8.

[0148] The figure 910 shows the original data to be calculated (which can be neurons or weights), and its storage order is HWC. The figure also shows 4 data blocks 911-914 in which the original data to be calculated is split according to the splitting unit, and each data block includes 4×4×4=64 data.

[0149] Figure 920 shows the format of the split data for easy reading. It can be seen that the original data blocks (e.g., 911-914) are arranged into a row (e.g., 921-924) on the C dimension. In each row, the data is stored in the order of CHW. For example, for data row 921, 16 data with C=0 are stored first, followed by 16 data with C=1, then 16 data with C=2, and finally 16 data with C=3.

[0150] Specifically, for neurons, the data needs to be arranged from [1Hi Wi Ci] to:

[0151] [1×Hi / 4×Wi / 4×Ci / 4×(4×4×4)], the shape of this seven-dimensional tensor.

[0152] For weights, the data needs to be arranged from [Co Kh Kw Ci] to:

[0153] [Co×Kh / 4×Kw / 4×Ci / 4×(4×4×4)], the shape of this seven-dimensional tensor.

[0154] As can be seen from the above description, the Forward4 scheme can support multiple grouping modes. For neurons, according to different grouping modes and HoWo splitting methods within the group, the seven-dimensional shape of the above block format is finally split into each storage area of ​​the second storage circuit with slight differences.

[0155] Assume that the original input neuron size is: [1×hi×wi×ci]

[0156] In Group 1 mode, the number of input neurons varies depending on the HoWo splitting method:

[0157] Ho×Wo 4×4 split: 16 [hi / (4×4), wi / (4×4), ci / 4×(4×4×4)]

[0158] Ho×Wo 1×16 split: 16[hi / (4),wi / (4×4×4),ci / 4×(4×4×4)]

[0159] In the above 4×4 split, 16 represents 16 slave processing circuits SL, and the last 4×4×4 (CHW) represents the BLOCK of CHW split from three dimensions. In the two 4s of hi and wi, the first 4 represents splitting hi×wi into 16 parts and distributing them to 16 SLs, and the second 4 represents folding hi and wi to ci. The meaning of 1×16 split is the same.

[0160] In Group 4 grouping mode, the number of input neurons varies depending on the HoWo splitting method:

[0161] Ho×Wo1×4 split: 4×4×[hi / (1×4),wi / (4×4),ci / 4×(4×4×4)]

[0162] For a slave processing circuit SL: hi / (1×4), wi / (4×4), ci / 4×(4×4×4)

[0163] In the above representation, the first 4 indicates 4 SLBs, the neuron is copied 4 times, the second 4 indicates the neuron is split on 4 SLs of one SLB, and the last 4×4×4 indicates the BLOCK of CHW split from three dimensions. In Group16 grouping mode, the input neuron does not need to be split, and its number of placements is as follows:

[0164] 16×[hi / 4,wi / 4,ci / 4×(4×4×4)]

[0165] The above 16 indicates that the neuron is replicated on 16 SLs. The 4×4×4 at the end indicates the BLOCK of CHW split from three dimensions. Both hi and wi are divided by 4, which means folding hi and wi to ci direction.

[0166] Output point splitting between operation circuits in Forward4 scheme

[0167] When multiple computing circuits CU in a single slave processing circuit SL jointly process a Co value, the output points need to be split among the multiple CUs.

[0168] Fig.10 FIG. 4 is a schematic diagram showing the allocation of interval output points to each operation circuit in the Forward4 scheme according to some embodiments of the present disclosure. In these embodiments, the N CU The output feature block is evenly divided into Nop output feature sub-blocks of the same shape between the operation circuits, and each output feature sub-block includes N CU output points, divided into N CUoperation circuits. For example, the figure takes the example that each SL includes 4 CUs and each CU can calculate Nop=4 output points or partial sums at a time, and shows that the output feature block 1010 includes 4×4 output points, and each output feature sub-block 1011~1014 divided evenly includes 2×2 output points. In each output feature sub-block, these 2×2 output points are assigned to 4 operation circuits. Thus, each operation circuit calculates one output point in each of the 4 output feature sub-blocks. The output points assigned to 4 different operation circuits CU0~CU3 are shown in different backgrounds in the figure. It can be seen from the figure that in each calculation, each operation circuit calculates multiple output points spaced in the X and / or Y dimensions on the output feature map.

[0169] Based on the above output point division, when the convolution operation is performed by sliding selection, N can be selected from the first buffer circuit corresponding to the output point position of each output feature sub-block according to the data required for calculating the output feature sub-block. CU For example, when the input feature data is first selected, 4 input data rows can be selected from the corresponding input feature blocks according to the 4 input feature blocks required for calculating the 4 output points in the first output feature sub-block 1011, and distributed to 4 operation circuits. It can be understood that since these 4 output points are continuous in the X and / or Y direction, the interval or step size of the 4 input data rows selected at the same time in the X and / or Y direction is 1.

[0170] When selecting weight data, the corresponding weight data can be selected from the second buffer circuit and broadcast to N CU A plurality of operation circuits are provided, thereby realizing parallel calculation of output points corresponding to the plurality of operation circuits by multiplexing weight data. Further, in some embodiments, in order to give full play to the computing power (e.g., multiplication and addition operator) inside the operation device CU, for example, to calculate Nop output points or partial sums at a time, weight multiplexing can be performed within a single input data row, thereby calculating Nop output points or partial sums at the same time.

[0171] For example, when selecting weight data, we can take only 1 / Nop weight rows, copy them Nop-1 times to expand them into 1 weight row, and this expanded weight row includes Nop identical 1 / Nop weight rows. The expanded weight row can also be broadcast to N CU A plurality of operation circuits are provided, so that while the weights are multiplexed between the plurality of operation circuits, the weights are multiplexed with a smaller granularity (eg, 1 / Nop row) between the calculations of the Nop output points of a single operation circuit.

[0172] Therefore, by taking N CU Input feature data rows, take 1 / Nop weight rows and copy them to expand them into 1 weight row, and N weights can be calculated each time. CU×Nop output points or partial sums. When the calculation result is a partial sum, the partial sum can be calculated multiple times by sliding multiple times, and the partial sums of each time are accumulated according to the output points to which they belong to, and the final result can be obtained.

[0173] According to the way the output points are divided, the number of sliding times and sliding step size of the convolution operation can be determined. Fig.10 The partitioning method is as follows: the number of slides Nk = ceil(Kx / 2) × ceil(Ky / 2), where Kx and Ky are the smaller values ​​of the size of the convolution kernel in the X and Y dimensions and the maximum convolution kernel size supported by the slave processing circuit in a single operation in the current convolution splitting mode, and the sliding step size = 2. The maximum convolution kernel size supported by the slave processing circuit in a single operation is, for example, determined by at least the spatial size of the first buffer circuit and the second buffer circuit. It can be understood that when the convolution kernel exceeds the maximum convolution kernel size, it needs to be split according to the maximum convolution kernel size in the Kx and Ky directions.

[0174] Convolution sliding process in the Forward4 scheme

[0175] Fig.11 A schematic diagram of a single operation process in the Forward4 scheme according to an embodiment of the present disclosure is shown. In this example, the size of the first buffer circuit 1110 is 3×3×64B, that is, it can cache up to 9 lines of data, and the size of the second buffer circuit 1120 is 2×2×64B, that is, it can cache up to 4 lines of data. In order to be consistent with the split unit, the storage in the buffer circuit in the figure is also shown in units of split units.

[0176] The figure shows the operation process of the first sliding data acquisition. According to the method corresponding to the division method of the output point, the split unit is used as a sliding window to slide and select N from the first buffer circuit. CU Input feature rows are sent to N CU arithmetic circuits for calculation; select 1 / Nop weight rows from the second buffer circuit in a sliding manner corresponding to that in the first buffer circuit, where Nop is the maximum number of convolution output points that can be calculated at a time for each arithmetic circuit, and copy Nop-1 copies to expand it into an extended weight row, which is broadcasted to N slave processing circuits; CU An operational circuit.

[0177] Specifically, in Figure 5 In the computing device shown, N CU =4, Nop=4. When dividing the output points, each operation circuit calculates 2×2 output points with an interval of 1 in both the X and Y dimensions for division in each calculation.

[0178] As shown in the figure, one input feature data row is selected from the first buffer circuit 1110 at the starting position and the position shifted by 1 in the X and / or Y direction, and a total of 4 input feature data rows are selected and sent to the 4 operation circuits 1140 in the slave processing circuit SL accordingly. A 1 / 4 weight data row is selected from the second buffer circuit 1120 at the starting position, that is, data of size 2×2 is selected, which is replicated 3 times to expand into an extended weight data row 1130, and broadcasted to the 4 operation circuits 1140 in the SL.

[0179] In each calculation, each operation circuit performs bitwise multiplication and accumulation on an input feature row from the first buffer circuit and an extended weight row from the second buffer circuit in units of 1 / Nop data rows to obtain Nop partial sums.

[0180] As shown in the figure, the four operation circuits 1140 perform bitwise multiplication and accumulation operations on the distributed input feature data rows and the broadcasted extended weight data rows to obtain the operation result 1150. The results of different background colors in 1150 represent the results obtained by different operation circuits 1140. It can be seen that in each operation, a CU will calculate the partial sum of 4 output points, and the four CUs will obtain a total of 4×4 partial sums. It can be seen that the output points calculated by each CU are not adjacent in the XoYo dimension of the output feature map.

[0181] Next, the first buffer circuit and the second buffer circuit are synchronously sliding to fetch data and perform the next calculation. Perform Nk sliding selections, where Nk = ceil(Kx / 2) × ceil(Ky / 2), Kx and Ky are the smaller values ​​of the size of the convolution kernel in the X and Y dimensions or the maximum convolution kernel size supported by the processing circuit in a single operation in the current convolution split mode. Accordingly, the operation circuit accumulates the Nk×Nop parts and the corresponding convolution output points obtained in the Nk sliding calculations to obtain Nop operation results.

[0182] In some embodiments, in Forward4 mode, the maximum convolution kernel size supported by a single operation of the slave processing circuit is 8×8.

[0183] Fig.12A schematic diagram of the sliding convolution process in the Forward4 scheme according to an embodiment of the present disclosure is shown. This example takes a 9×9 input feature map and a 5×5 convolution kernel as an example. The convolution step size is 1, and the output feature map size is 5×5. The input feature map needs to be aligned to 12×12, divided into 9 blocks of 4×4×4 (C×H×W) in size, and stored in the first buffer circuit, which is shown as 1210 in the figure, and the C dimension is omitted. The convolution kernel 5×5 needs to be aligned to 8×8, and the aligned part is padded with 0s and stored in the second buffer circuit, which is shown as 1220 in the figure, and the C dimension is also omitted. In each calculation, a 2×2 block in the convolution kernel is selected and copied 4 times, which just corresponds to the 4×4 block of the input feature map. The copying operation can be implemented by hardware.

[0184] The input feature map and the convolution kernel selection range in the first buffer circuit and the second buffer circuit at each slide are as follows: Fig.12 As shown, there are 9 pictures in total, representing a total of 9 slides. In the figure, block 1210 represents the input feature map in the first buffer circuit, and the four dotted boxes represent the areas selected to be sent to four CUs; ​​block 1220 represents the convolution kernel in the second buffer circuit, and the dotted box represents the selected 1 / 4 line, which is copied 3 times and expanded into one line and then broadcast to 4 CUs. The number of slides Nk = ceil (Kx / 2) × ceil (Ky / 2) = 9.

[0185] During each calculation, each CU performs bitwise multiplication and accumulation on an input feature data row from the first buffer circuit and an extended weight data row from the second buffer circuit in units of 1 / 4 data rows to obtain 4 partial sums; and in the current operation round, accumulates the Nk partial sums corresponding to the same convolution output point obtained in Nk calculations to obtain and output 4 operation results.

[0186] Specifically, for Fig.12 For each image in the image, the number of CUs Ncu=4, and each CU calculates Nop=4 output points or partial sums at a time. The partial sum is the result of the bitwise multiplication and accumulation of 1 / 4 data rows, that is, each output point is a standard convolution of 4×2×2 (Ci×Y×X). After sliding Nk=ceil(Kx / 2)×ceil(Ky / 2)=9 times, the accumulation is completed in the Y×X direction, and finally a complete 4×4 (Y×X) output is obtained in 1 SL. In this mode, a single calculation only supports the case where the convolution kernel is not larger than 8×8. For larger convolution kernels, it is necessary to split them into 8×8 in the Kx and Ky directions, and the splitting operation can be performed according to the same principle as above.

[0187] It can be understood that when Ci>4, it is necessary to traverse in the Ci direction and switch the input and weights at the same time until the complete output is calculated. When the Xo / Yo calculated by each CU is greater than 4, it is necessary to slide along the Xo / Yo direction to read different input neurons and weights. Those skilled in the art can deduce the calculation process similarly based on the above description, which will not be repeated here.

[0188] Output shape description in Forward4 scheme

[0189] It can be seen from the previous output point division method and sliding convolution process that the result of the sliding mode output is not the normal arrangement order of the traditional convolution output data. Therefore, during the output process, each slave processing circuit SL can convert the operation result of its internal operation circuit CU into a specified format, such as the format of Nco×Uy×Ux. In some embodiments, each slave processing circuit can output partial operation results of its internal partial operation circuit each time, and the partial operation results are continuous in the X and / or Y dimensions of the output feature map. The main processing circuit can further store the operation results returned from each slave processing circuit in a fourth dimensional storage order. Depending on the situation, the main processing circuit can also convert the operation results into the desired dimensional storage order for storage.

[0190] When the grouping mode and / or the splitting method of the input feature map within a single SLB (that is, the splitting method according to the HoWo of the output feature map) is different, the output data format is slightly different.

[0191] Fig.13 The output data format of the Forward4 solution according to an embodiment of the present disclosure is shown in FIG. 1. In this embodiment, the grouping mode is Group1, and the input feature map in a single SLB (including 16 SLs) is split according to Ho×Wo=1×16.

[0192] 1310 in the figure shows the original output of one SL. As can be seen from the figure, each SL outputs a 1×1×4 (Co×Y×X) area each time, that is, it outputs part of the calculation results of its internal calculation circuits each time, for example, 2 calculation results of each of the 2 CUs (see Fig.10 ), this part of the operation results are continuous in the X and / or Y dimensions of the output feature map, for example, in the same row ( Fig.13 4 times in a row, the 1×4×4 (Co×Y×X) region is returned, which is the 4 operation results of each of the 4 CUs. Different SLs output different regions of the output feature map of the same Co. After all 4×4 regions of Co are output, the output will switch to different output points.

[0193] Figure 1320 shows the data structure of 16 SLs. As shown in the figure, the final output data becomes Yo×Xo×Co×4×16×4 format after being written into the storage circuit (e.g., the first storage circuit), where Yo and Xo are the number of blocks of the output feature map to which each SL is divided, and 16 is the division on 16 SLs. As needed, in some implementations, the swing operation can be performed again to convert into other desired data formats.

[0194] As mentioned earlier, the output data format may be slightly different depending on the grouping mode and / or the way the input feature map is split between multiple SLs within a single SLB. Assuming the original output size is:

[0195] 1×ho×wo×co

[0196] Then, the output data shape of Group1 when Ho×Wo is split into 4×4 is:

[0197] ho / (4×4)×wo / (4×4)×co / group×(4×16×4)

[0198] In the above formula, (4×16×4) is the basic output block of forward4, and the directions correspond to h×c×w, where 16 represents the division of ho and wo of the same co on 16 SLs, which can be specifically decomposed into 4[high-dimensional ho]×4[low-dimensional ho]×4[high-dimensional wo]×4[low-dimensional wo]; ho and wo are divided by 4 twice, where the first 4 represents the 4×4 split when storing data in SL, and the second 4 represents the folding of data blocks in the h and w directions. In Group1 mode, the above group=1.

[0199] The output data shape of Group1 when Ho×Wo is split according to 1×16 is:

[0200] ho / (4)×wo / (4×16)×co / group×(4×16×4)

[0201] In the above formula, (4×16×4) is the basic output block of forward4, and the directions correspond to h×c×w, where 16 represents the division of ho and wo of the same co on 16 SLs, which can be specifically decomposed into 4[low-dimensional ho]×16[high-dimensional wo]×4[low-dimensional wo]; in Group1 mode, the above group=1. This shape is Fig.13 The shape of the diagram.

[0202] It can be seen that in the case of Group 1, 16 SLs divide the Yo×Xo dimension of an output feature map equally. The data in the row dimension SL at output corresponds one-to-one to the way the 16 SLs divide the output neurons equally in the Yo×Xo direction. This scenario is suitable for input neurons with large values ​​in the Y×X direction and small Co values.

[0203] The output data shape of Group4 when Ho×Wo is split into 2×2 is:

[0204] ho / (2×4)×wo / (2×4)×co / group×(4×16×4)

[0205] In the above formula, (4×16×4) has the same meaning as above, except that 16 represents the division of wo outputs of 4 co on 4 SL, which can be specifically decomposed into 4[co]×4[high-dimensional ho]×2[low-dimensional ho]×2[high-dimensional wo]×4[low-dimensional wo]. In Group4 mode, the above group=4.

[0206] The output data shape of Group4 when Ho×Wo is split into 1×4 is:

[0207] ho / (1×4)×wo / (4×4)×co / group×(4×16×4)

[0208] In the above formula, (4×16×4) has the same meaning as above, except that 16 represents the division of wo outputs of 4 co on 4 SL, which can be specifically decomposed into 4[co]×4[low-dimensional ho]×4[high-dimensional wo]×4[low-dimensional wo]. In Group4 mode, the above group=4.

[0209] The output data shape of Group16 is:

[0210] ho / 4×wo / 4×co / group×(4×16×4)

[0211] In the above, (4×16×4) has the same meaning as above, except that 16 represents the output division of 16 cos on 16 SLs, which can be specifically decomposed into 4[low-dimensional ho]×16[co]×4[low-dimensional wo]. In Group16 mode, the above group=16.

[0212] Since Group has different split categories in the H×W direction, the 16 in the 4×16×4 mentioned above has differences in specific splits. Since Forwrd4 is calculated based on 4B×4×4 blocks, it is inevitable that there are alignment restrictions during calculation. According to different Group modes, different H×W splitting methods of the same Group mode will eventually have different alignment restrictions during calculation. In the calculation of alignment, we can first determine the alignment restrictions of ho×wo according to the splitting method of the output feature map, and then reversely infer hi×wi from ho×wo. Since the input neurons need to be arranged in the form of split unit blocks, they need to be aligned again. The above alignment restrictions can be summarized in the following Table 3:

[0213]

[0214]

[0215] Table 3. Alignment constraints

[0216] In summary, when outputting, the hardware can automatically output neurons in the form of 4×16×4 (Y×SL×X) dimensions within a row and Y×X×C dimensions between rows. The same is true for larger convolution kernels.

[0217] Biased shape description in Forward4 scheme

[0218] Bias is the bias after the convolution calculation is completed. The original format of the bias is: [1 1co].

[0219] Since the data format of Forward4 output is ho×wo×co / group×(4×16×4), if you need to add an offset to the data directly output by Forward4 on the chip, you need to change the basic shape of the offset. The placement format of the offset in the on-chip space is related to the Group grouping mode. Specifically, the number of offsets in various grouping modes is as follows:

[0220] In Group 1 mode, the number of biased swings: [1 1co×64]

[0221] Here, 64 means that a single offset is copied 64 times and placed consecutively.

[0222] In Group 4 mode, the number of biased pendulums is: [1 1co×16]

[0223] Here, 16 means that a single offset is copied 16 times and placed consecutively.

[0224] In Group 16 grouping mode, the number of biased pendulums is: [1 1co×4]

[0225] Here, 4 means that a single offset is copied 4 times and placed consecutively.

[0226] Data handling process

[0227] From the previous description of the small convolution operation scheme, we can see that the input neurons and weights need to be split and stored in dimension transformation, and the output neurons also need to undergo certain dimension transformation. Figure 3b The hardware structure of the multi-core computing device, for the sake of hardware IO efficiency, the input data needs to be read from the global memory first, and then stored in the shared storage SRAM after loading the data. As mentioned earlier, Forward4 needs to split neurons. Taking into account the alignment factor, its splitting characteristics determine that Forward4 has more computational advantages when processing input feature maps that are relatively large and have a relatively small number of channels. Therefore, when it comes to the hardware design of Forward4, larger neurons can be stored in WRAM and relatively small weights can be stored in NRAM. At the same time, since the weights and neuron data need to be arranged in the form of blocks described above, the neurons stored in WRAM also need to pass through NRAM once to transform the shape of the tensor data.

[0228] Fig.14 The overall data handling process according to the embodiment of the present disclosure is shown.

[0229] As shown in the figure, the weights are read from off-chip storage, such as DDR, into SRAM via the global direct memory access module (GDMA). HW plane alignment and padding operations are completed on SRAM. Tiling instructions are used in the process of data from SRAM to NRAM, which can complete the data transfer process as well as the data dimension transformation and alignment process.

[0230] The process of transferring neurons is similar to that of weights, except that after being transferred to NRAM through block instructions, they also need to be transferred to WRAM. Since most of the data overlaps as the convolution kernel slides during neuron calculation, this greatly reduces the efficiency of data transfer. In order to solve this problem, the img2col instruction is used in some embodiments of the present disclosure to distribute data, as described in detail below.

[0231] The output data can be stored back to NRAM, and the data dimension can also be changed and moved to SRAM through block instructions. Then, it can be stored back to the off-chip DDR storage via GDMA.

[0232] Exemplary principle of block instruction

[0233] Data dimension change and placement refers to the process of placing tensor data of a specific shape into the required specific shape. Data handling refers to the reading and writing operations of data in different memory spaces. As mentioned above, the Forward4 convolution operation scheme requires that the neurons and weights used for convolution operations are arranged and aligned in a specific block mode. In addition, the output data is also output in the specific output format of Forward4, which requires that the tensor data be arranged in block form before calculation, and also requires that it be placed back to the normal tensor shape as required after the calculation is completed.

[0234] In the disclosed embodiment, in the process of transferring the input neurons, weights, and bias data from SRAM to NRAM, and in the process of transferring the output data from NRAM to SRAM, the block instruction (Trans Tiling) is used to complete the transfer operation. In this transfer process, it is necessary to complete the basic data transfer process, and it is also necessary to complete the data dimension change and placement process to meet the calculation requirements.

[0235] The Deform instruction series provides the IO data path with the ability to transform data shapes and convert data types, mainly including TRANS (transposition), MOVE (movement), ROTATE (rotation) and other functions. The mode for implementing the transposition function in this instruction series is named Trans Tiling, which mainly provides performance support for various shape transformations of small convolutions. Deform divides a 3D data block into two layers, the inner and outer layers. The inner layer has three dimensions (corresponding to the parameters n0-2 in the instruction). The unit of the lowest dimension is byte, and the second lowest and highest dimensions are unitless, representing the number of the previous layer. The outer layer also has three dimensions (corresponding to the parameters n3-n5 in the instruction), all representing multiples of the corresponding inner layer dimensions.

[0236] When implementing the small convolution splitting scheme, the input data stored in the first dimensional storage order (such as HWC) needs to be split, dimensionally converted and stored in split units. Each split unit is stored in the second dimensional storage order (such as CHW), and the split units are stored in the third dimensional storage order (such as HWC).

[0237] Fig.15 A schematic conceptual diagram of Trans Tiling according to an embodiment of the present disclosure is shown.

[0238] The left figure shows the input data before deformation. It can be seen that the three-dimensional input data is described using six dimensions, n0 and n3 correspond to the first dimension of the original three-dimensional data (e.g., the lowest dimension), n1 and n4 correspond to the second dimension of the original three-dimensional data (e.g., the second lowest dimension), and n2 and n5 correspond to the third dimension of the data block (e.g., the highest dimension). In the example in the figure, the inner layer of the input data corresponds to the split unit. Taking the Forward4 scheme as an example, the inner layer data block of the input data is a 4B×4×4 data block, where n0=4B, n1=n2=4.

[0239] The right figure shows the deformed output data. The three-dimensional output data is also described using six dimensions. At this time, the inner layer of the output data corresponds to the deformed split unit. In the Forward scheme, the inner layer data block of the output data is a 64B×1×1 data block, where n0=64B, n1=n2=1.

[0240] In addition, Trans Tiling also has an inline shuffle function, including inline shuffle before tiling based on the pretable, and inline shuffle after tiling based on the posttable. The pretable is a function for rearranging the data of n0 of the tiling input, and the posttable is a function for rearranging the data of n0 of the tiling output. Without considering the flag bit of the table, the pretable and posttable are essentially an array representing the position of 64 bytes of data.

[0241] Fig.16 A schematic diagram of the front and rear matching table is shown.

[0242] As shown in the figure, the front and back tables respectively represent the rearrangement position of a row of data of the n0 dimension of the input or output, which includes 64B. The 8 bits of each byte include 6 bits of index bits, which record the data of the byte number of the 0-63 byte data in the original data; 1 bit of zero_en bit, indicating whether it is set to 0. If this bit is 1, it is forced to write 0, and the [5,0] bit is invalid; and 1 bit of mask bit, indicating whether this bit of data is valid.

[0243] By using the front and back tables, the data of n0 of the input data of the block instruction can be rearranged when necessary, and / or the data of n0 of the output data of the block instruction can be rearranged.

[0244] Table 4 shows the meaning of each parameter of the block instruction. Assuming that the bit width of the data to be divided into blocks is dwidth, the unit is B (byte), the amount of data atomically operated by the block instruction is called the block bit width T, the unit is B (byte). Among the parameters of the block instruction, 11 parameters, n0~n5, s1~s5, are needed to describe the tensor shape of the inner and outer data, among which n0~n2, s1~s2 are the parameters describing the inner layer, and n3~n5, s3~s5 are the parameters describing the outer layer.

[0245]

[0246] Table 4. Meaning of parameters of block instructions

[0247] For the tensor description before and after the execution of the block instruction, the input tensor and the output tensor each require a set of parameters, which are described by 22 parameters in total, namely in0~in5, is1~is5, on0~on5, and os1~os5. The block instruction can support a variety of block widths T, such as 1B, 2B, 4B, 6B, 8B, 16B, 32B, etc. Based on different block tasks, the corresponding values ​​can be set. Therefore, the block width T parameter is also included in the block instruction.

[0248] When using block instructions, there are some basic restrictions or constraints. These restrictions include, for example: in0, in1, in2, on0, on1, on2<=64; n0 performance requires 64B alignment; in0=on1×on2×T, on0=in1×in2×T; in3×in4×in5=on3×on4×on5; T<=32B; front and back table = 64B.

[0249] In addition, the block instruction cannot be operated in situ, that is, two storage areas are required. Therefore, in the disclosed embodiment, a data processing device is provided, including a control circuit, a first storage circuit, and a second storage circuit. The first storage circuit is used to store the first data before the block instruction is executed; the second storage circuit is used to store the second data after the block instruction is executed. The control circuit is used to configure and execute the block instruction. In some embodiments, the data processing device can be, for example, Figure 3b In the processor cluster in the multi-core computing device shown, the control circuit is, for example, a processor core in the processor cluster, the first storage circuit is, for example, a shared storage SRAM in the processor cluster or an NRAM in the processor core, and the second storage circuit is, for example, an NRAM or an SRAM. When executing block instructions for different data (input neurons, weights, output neurons, biases, etc.), the dimensional changes and handling processes that need to be implemented are also different, and different block instruction parameter configuration schemes need to be designed.

[0250] Block instruction scheme for bias data

[0251] According to the description of small convolution operation schemes such as Forward4 in the previous article, when Forward4 adopts different Group grouping modes and / or different Ho×Wo splitting methods within the group, the format of the output data (that is, the output neuron) is not exactly the same. The detailed description of the output data format can be found in the output shape description section of the Forward4 scheme in the previous article. These output data formats can be simplified to the ho×wo×co / group×(4×16×4) format.

[0252] The convolution operation also includes bias data, which is superimposed on the convolution output result. Its initial shape is [1*1*co]. If you need to add a bias to the data directly output by Forward4 on the chip, you need to change the basic shape of the bias to match the output data shape of Forward4.

[0253] As mentioned in the bias shape section above, the placement format of the bias in the on-chip space is related to the grouping mode. Specifically, the number of bias placements in various grouping modes is as follows:

[0254] In Group 1 mode, the number of biased swings is: [1 1co×64],

[0255] Here, 64 means that a single offset is copied 64 times and placed consecutively.

[0256] In Group 4 grouping mode, the number of biased pendulums is: [1 1co×16],

[0257] Here, 16 means that a single offset is copied 16 times and placed consecutively.

[0258] In Group 16 grouping mode, the number of biased pendulums is: [1 1co×4],

[0259] Here, 4 means that a single offset is copied 4 times and placed consecutively.

[0260] Assuming group = {1, 4, 16}, the final bias data shape can be expressed as:

[0261] From the above deformation and pendulum number requirements for the bias data, it can be seen that the above process only copies each number in the co dimension If this copy process is completed using a scalar copy operation, co × 64 instruction operations are required, and the implementation performance is relatively poor.

[0262] In the disclosed embodiment, the block instruction is used to implement the above-mentioned offset data copying and placement process. More specifically, the pre-matching table function of the block instruction can be used to implement the data copying operation.

[0263] Fig.17 FIG. 4 is a schematic diagram of block processing of bias data according to an embodiment of the present disclosure. In this example, it is assumed that the Forward4 convolution operation scheme adopts the Group1 grouping mode.

[0264] The left figure shows the bias data before block processing (that is, the input tensor of the block instruction). It can be seen that the three-dimensional bias data [1×1×co] can be expressed as a three-dimensional equivalent shape [1×co×1]. The dotted line shows the dimensional expansion caused by copying co 64 times. The expanded bias data is divided into two layers, inner and outer, each described using three dimensions. Since the maximum block width T is 32B, the in0 dimension of the inner data block 1701 can be set to min(64 / group×dwidth,32B); the in1 dimension and the in2 dimension are both set to 1. The block width is also set to in0, so that one inner data block can be processed at a time. After determining the inner data block, the sizes of the three outer dimensions in3, in4 and in5 can also be determined accordingly, and their sizes are equal to the number of inner data blocks contained in the corresponding dimensions.

[0265] Note here that since the input tensor of the block instruction is read from the first storage circuit (such as SRAM) and then copied, the dimension step is3 of in3 is set to 0.

[0266] The right figure in the figure shows the bias data after block processing (that is, the output tensor of the block instruction). At this time, the shape of the bias data becomes [1×1×co×64], where each data in the co dimension is copied 64 times. It can be seen that compared with the bias data before block processing, the bias data after block processing only has the co dimension copied, and the other dimensions have not changed. As shown in the figure, the inner data block 1702 of the output tensor is the same as the inner data block 1701 of the input tensor, and the overall shape is also the same as the input tensor after dimensional expansion. Therefore, the six dimensional sizes of the output tensor are exactly the same as the six dimensional sizes of the input tensor. The difference is that since co×64 is obtained by copying in the output tensor, these copied data will be stored in the second storage circuit, so the dimension step os3 of the output tensor on3 is no longer 0, but equal to min(64 / group×dwidth,32B).

[0267] Table 5 shows the shape change of the offset data before and after executing the block instruction.

[0268]

[0269]

[0270] Table 5. Shape changes of bias data before and after block processing In an example, the block instructions for the bias data can be configured according to the following Table 6.

[0271]

[0272] Table 6. Parameters for offset data block instructions

[0273] Wherein, dwidth represents the data bit width, and the unit is B, where B represents byte; T represents the block bit width, and the unit is also B, which represents the amount of data for an atomic operation of the block instruction at one time; in0, in1, and in2 respectively represent the inner lowest dimension size, the inner low dimension size, and the inner highest dimension size of the inner data block of the input tensor of the block instruction; in3, in4, and in5 respectively represent the three outer dimension sizes of the input tensor, and the size values ​​of the three outer dimensions respectively represent the number of the inner data blocks contained in the corresponding dimensions; is1~is5 represent the five dimensional steps of the input tensor except the inner lowest dimension; on0, on1, and on2 respectively represent the inner lowest dimension size, the inner low dimension size, and the inner highest dimension size of the inner data block of the output tensor of the block instruction; on3, on4, and on5 respectively represent the three outer dimension sizes of the output tensor, and the size values ​​of the three outer dimensions respectively represent the number of the inner data blocks contained in the corresponding dimensions; os1~os5 represent the five dimensional steps of the output tensor except the inner lowest dimension.

[0274] It can be seen from the configuration scheme in Table 6 that the maximum block width T is 32B (that is, 256 bits). Since n1 and n2 are 1, the upper limit of n0 is 32B.

[0275] The pre-matching table only needs to implement the copy function. In some embodiments, the control circuit can configure the block instruction as follows: set the pre-matching table of the block instruction, which is used to copy the data of the co dimension min (64 / group, 32B / dwidth) times as the inner lowest dimensional data of the input tensor of the block instruction, where dwidth represents the data bit width.

[0276] In one implementation, the pre-configuration table may be configured according to the logic shown in the pseudo code of Table 7 below.

[0277]

[0278]

[0279] Table 7. Pseudocode for tabulation before offset data block instruction

[0280] In addition, combined Fig.17 It can also be seen from the parameter configuration scheme in Table 6 that the six dimensional sizes of the input tensor of the block instruction and the six dimensional sizes of the output tensor are set to be the same, that is, the inner data block and outer size of the input tensor and the output tensor are the same. Fig.17 The dotted part in the left figure) is realized by the copy function of the previous configuration table.

[0281] Therefore, the above describes the block processing scheme of bias data in conjunction with the specific example of the Forward4 scheme. By using block instructions to implement the copying and swinging of bias data, the on-chip addition processing of bias data and output neurons can be realized, while also improving the processing efficiency.

[0282] The disclosed embodiment also provides a data processing method for executing block instructions using the aforementioned data processing device. Those skilled in the art can understand that the method steps for executing block instructions correspond to the various features of the computing device described above in conjunction with the accompanying drawings, so the features described above are also applicable to the method steps and will not be repeated here.

[0283] The present disclosure also provides a chip, which may include the data processing device of any of the embodiments described above in conjunction with the accompanying drawings. Further, the present disclosure also provides a board, which may include the aforementioned chip.

[0284] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server computing clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0285] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0286] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other splitting methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0287] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0288] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0289] The embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A data processing device, comprising a control circuit, a first storage circuit and a second storage circuit, wherein: The first storage circuit is used to store the first data before processing; The second storage circuit is used to store the processed second data; as well as The control circuit is used to configure and execute a block instruction to convert first data stored in a first storage circuit according to a first dimensional storage order into second data stored in a second storage circuit according to a second dimensional storage order, wherein the first data is three-dimensional data, and its three-dimensional shape is: [1×1×co] The second data is also three-dimensional data, and its three-dimensional shape is: Where co represents the output channel dimension of the data, group represents the number of groups in the grouping mode in the associated convolution operation, Indicates that each data in co is copied Second-rate.

2. The data processing apparatus according to claim 1, wherein the control circuit is further configured to: Treat the first data from the three-dimensional shape as a three-dimensional equivalent shape before the block processing: [1×co×1] The highest dimension is 1, the second highest dimension is co, and the lowest dimension is 1.

3. The data processing apparatus according to claim 2, wherein the control circuit is further configured to configure the block instruction as follows: A pre-matching table for the block instruction is set, and the pre-matching table is used to copy the data of the co dimension min(64 / group, 32B / dwidth) times as the inner lowest dimensional data of the input tensor of the block instruction, where dwidth represents the data bit width.

4. The data processing apparatus according to claim 3, wherein the control circuit is further configured to configure the block instruction as follows: The six dimensional sizes of the input tensor and the six dimensional sizes of the output tensor of the block instruction are set to be correspondingly the same.

5. The data processing device according to claim 4, wherein the control circuit is further configured to configure the block instruction according to the following Table 1: Table 1 Wherein, dwidth represents the data bit width, B represents bytes, T represents the block bit width, which represents the amount of data for one atomic operation of the block instruction, in0, in1, in2 represent the inner lowest dimension size, inner low dimension size and inner highest dimension size of the inner data block of the input tensor of the block instruction respectively, in3, in4 and in5 represent the three outer dimension sizes of the input tensor respectively, and the size values ​​of the three outer dimensions respectively represent the number of the inner data blocks contained in the corresponding dimensions, is1~is5 represent the five dimensional steps of the input tensor except the inner lowest dimension, on0, on1, on2 represent the inner lowest dimension size, inner low dimension size and inner highest dimension size of the inner data block of the output tensor of the block instruction respectively, on3, on4 and on5 represent the three outer dimension sizes of the output tensor respectively, and the size values ​​of the three outer dimensions respectively represent the number of the inner data blocks contained in the corresponding dimensions, and os1~os5 represent the five dimensional steps of the output tensor except the inner lowest dimension. The data processing apparatus according to claim 5 , wherein the group={1, 4, 16}.

7. A data processing device according to any one of claims 1-6, wherein the first data is bias data, which is used to be added to the output neuron data obtained by using the Forward4 small convolution operation scheme.

8. A chip, comprising the data processing device according to any one of claims 1-7.

9. A board comprising the chip according to claim 8.

10. A data processing method using the data processing device according to any one of claims 1 to 7 to execute a block instruction on offset data.

Images