Method and apparatus for generating architecture-specific convolution gradient kernels

Abstract

A method for accelerating a convolution operation includes receiving a first data set and a second data set from an I / O interface. Converting the first data set into a first transformed data set, the first transformed data set having a first format. Converting the second data set into a second transformed data set, the second transformed data set having a second format. Loading the first transformed data set and the second transformed data set into a convolution function unit, wherein the convolution function unit is configured to receive first data in a first format, receive second data in a second format, and output third data in a third format. A task scheduler receives a result in the third format from the convolution function unit.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a method and apparatus for determining parameters of a convolution operator for training a neural network. Background Art

[0002] Convolutional neural networks (CNNs) are a class of neural networks commonly used for visual image analysis. In a CNN, a convolution operation is applied to a dataset and a set of weights. Convolution is one of the most resource-intensive mathematical operators used in modern neural networks, and considerable work has been done to optimize it. Numerous schemes have been proposed, many of which use common routines such as the Fast Fourier Transform (FFT) or the image to column (Im2Col) transform. FFTs require a large amount of memory, making them difficult to execute on systems with limited memory or memory bandwidth. The Im2Col routine, on the other hand, converts convolution into an equivalent general matrix multiplication (GEMM)-based operation. The advantage is that matrix multiplication operations are widely used and are often highly optimized for the chosen hardware and processor architecture.

[0003] A CNN takes in a set of data inputs and associated weights. Before it can be used, the neural network must be trained using the input and output sets. The training operation consists of a forward pass, which processes the inputs and weights, and a backward pass, which processes the gradients and updates the weights.

[0004] When the neural network is in the backward pass of training mode, the inverse of the convolution operation must be used. Since the forward pass of the convolution can use the Im2Col routine, the backward pass usually uses the inverse of the Im2Col routine, the column to image (Col2Im) routine. In Col2Im, the gradients are scattered from the matrix back to the corresponding components of the input used in the forward pass. In many computing architectures, the Col2Im function includes a large number of unoptimized vector operations, resulting in slow computation. Since training a CNN may require a large number of backward passes, this may significantly delay the development of optimized neural networks.

[0005] What is needed is a method and apparatus for back-propagating convolution operations that does not suffer from one or more limitations of the prior art.

[0006] This background information is provided to reveal information believed by the applicant to be potentially relevant to the present invention. It is not necessarily intended, nor should it be construed, that any of the foregoing information constitutes prior art against the present invention. Summary of the Invention

[0007] An embodiment of the present invention aims to provide a method and apparatus for accelerating convolution operations, which can be used for training neural networks. According to an embodiment of the present invention, a system for accelerating convolution operations is provided, comprising: a task scheduler; a plurality of cores, each of the plurality of cores comprising a convolution functional unit, each of the convolution functional units configured to receive first data in a first format, receive second data in a second format, and output third data in a third format. The system further comprises: an I / O interface coupled to the task scheduler; and a non-transitory computer-readable storage medium storing software instructions for controlling the task scheduler. The task scheduler performs the steps of receiving a first data set and a second data set from the I / O interface; transforming the first data set into a first transformed data set having the first format; transforming the second data set into a second transformed data set having the second format; loading the first transformed data set and the second transformed data set into the convolution functional unit; and receiving a result in the third format from the convolution functional unit.

[0008] In another embodiment, the software instructions further control the task scheduler to execute the step of transforming the result into a conversion result.

[0009] In another embodiment, the result is in a 5D format and the converted result is in a fractal format.

[0010] In another embodiment, the first data set is a data set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

[0011] In another embodiment, the first data set is in 5D format, the first transformed data set is in 5D format, the second data set is in 5D format, and the second transformed data set is in fractal format.

[0012] In further embodiments, the result is a gradient with respect to the set of weights.

[0013] In another embodiment, the first data set is a set of weights of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

[0014] In another embodiment, the first data set is in a fractal format, the first transformed data set is in a fractal format, the second data set is in a 5D format, and the second transformed data set is in a 5D format.

[0015] In further embodiments, the result is a gradient relative to the data set.

[0016] In further embodiments, said gradient relative to said dataset is in a 5D format.

[0017] In some embodiments, the system further comprises: a retargetable data converter, wherein said transforming said first data set into a first transformed data set is performed by a vector or scalar functional unit on a core running said retargetable data converter to perform said transform.

[0018] In another embodiment, the convolution function processes a three-dimensional matrix.

[0019] In another embodiment, the convolution functional unit performs an img2col function followed by a matrix multiplication function.

[0020] In further embodiments, the first data set and the second data set are in tensor format.

[0021] According to another aspect, an embodiment includes a method for accelerating a convolution operation. The method includes: a task scheduler receiving a first data set and a second data set from an I / O interface. The task scheduler transforms the first data set into a first transformed data set, the first transformed data set having a first format. The task scheduler transforms the second data set into a second transformed data set, the second transformed data set having a second format. The task scheduler loads the first transformed data set and the second transformed data set into a convolution functional unit, the convolution functional unit being configured to receive first data in a first format, receive second data in a second format, and output third data in a third format. The task scheduler receives a result in the third format from the convolution functional unit.

[0022] Another aspect of the present invention includes a method for transforming a tensor. The method includes receiving an input tensor having an input format having an input iterator set, and storing the input tensor in an input memory. Allocating output memory to store an output tensor having an output format having an output iterator set. Iterating over the output memory to copy input components of the input tensor to a portion of the output memory corresponding to output components of the output tensor, the input components having indices defined as functions of the output iterator set.

[0023] In some embodiments, the method further includes: allocating a chain memory to store a chain tensor having a chain format, the chain format having a chain iterator set; iterating on the chain memory to copy the output component of the output tensor to a portion of the chain memory corresponding to the chain component of the chain tensor, the index of the output component being defined as a second function of the chain iterator set.

[0024] Additional embodiments include: for the output tensor Y[i], the output format has the form (HW, 1), the output iterator set is i, for the input tensor X[j], the input format has the form (H, W), the input iterator set is j0, j1, and the function is Y[i] = X[i / W, i%W].

[0025] Additional embodiments include: for an output tensor Y[i], the output format has the form (CHW, 1), the output iterator set is i, for an input tensor X[j], the input format has the form (C, H, W), the input iterator set is j0, j1, j2, the function is

[0026] Additional embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has the form (HW, 1), the input iterator set is j0, and the function is Y[i0, i1] = X[i0*W+i1].

[0027] Additional embodiments include: for an output tensor Y[i], the output format has the form (C, H, W), the output iterator set is i0, i1, i2, for an input tensor X[j], the input format has the form (CHW, 1), the input iterator set is j0, the function is Y[i0, i 1, i2]=X[i0*HW+i1*W+i2].

[0028] Further embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has the form (H, W), the input iterator set is j0, j1, and the function is Y[i0, i1] = X[H–1–i0, W–1–i1].

[0029] Additional embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has (H, W), the input iterator set is j0, j1, and the function is Y[i] = X[i1, i0]. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features and advantages of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0031] Figure 1shows a simplified view of a system for inverse convolution;

[0032] Figure 2 A deconvolution system is shown with the data format required by the tensor processor having a fractal data format;

[0033] Figure 3 A deconvolution system is shown with the data format required by a tensor processor having a fractal data format, where the convolution output undergoes further transformation;

[0034] Figure 4 A system for accelerating the execution of a deconvolution operation is shown;

[0035] Figure 5 An example of a fractal data format provided by an embodiment is shown;

[0036] Figure 6 Configurations of computing devices that may be used with or in connection with the embodiments are shown.

[0037] It should be noted that throughout the drawings, like features are identified by like reference numerals. DETAILED DESCRIPTION

[0038] Embodiments of the present invention include systems and methods for accelerating convolution operations. This can utilize dedicated hardware, such as ASICs, coprocessors, cards, modules, or similar semiconductor or electronic devices known in the art. Embodiments of the present invention can be used to perform CNN training in AI systems.

[0039] Embodiments can be designed in various ways and, in the case of dedicated hardware resources, can be designed to support a limited number of input and output formats. Common formats used in the art include 4-dimensional (D), 5D, and fractal formats. In order for the hardware unit to process the data correctly, the input data must be in the correct format. In order for an external computing device to correctly interpret the output of the hardware unit, the external computing device must understand the output format.

[0040] In embodiments involving CNN training, a forward prediction operation is performed using a set of data and weights to produce an output. Both the input data and the weights will have their own formats, which may be determined by the source of the data. The output of the prediction operation will also have its own format. If hardware acceleration is used, the output format will be determined by the hardware design.

[0041] In most applications, prediction operations involve convolving data and weights using convolutional functional units. The convolutional layer is one of the most important layers in a deep neural network. There are many ways to implement a convolutional layer. A common and efficient approach is to convert the convolution into a general matrix multiplication (GEMM). This method performs an image-to-column (im2col) transformation to convert the image into a matrix, and then calls the GEMM function to perform the actual computation. This method is intensively optimized for performance. Some hardwired architectures and accelerators provide im2col processors and matrix multiplication units to accelerate convolution.

[0042] When training a CNN, the gradient of the convolution is calculated using the reverse convolution operation, which is usually much less optimized than the forward convolution operation. The common way to implement reverse convolution is to use the column-to-image (col2im) transformation, which is the inverse of im2col. col2im is much slower than im2col because it includes additional operations and usually lacks hardware acceleration.

[0043] Embodiments of the present invention reformat the input and output data so that a convolution operation can be used. In many embodiments, this convolution operation will use the same optimized convolution functional unit as the forward convolution operation described above.

[0044] The backward pass of the CNN training process involves determining the gradient of the loss with respect to (wrt) the input data and weights, and the gradient of the loss with respect to the output of the forward pass operation, given the head. The embodiment reformulates this calculation to use convolution operations that can take advantage of architectural optimizations.

[0045] For the case of no padding or dilation, provide X with X.shape = (H, W) and W with W.shape = (k h ,k w ) of 2D data convolution of W, where the stride is (s h ,s w ), is given by the following formula:

[0046]

[0047] The gradient of the output with respect to the data input is given by the following formula:

[0048]

[0049] The gradient of the output with respect to the weight input is given by the following formula:

[0050]

[0051] These two formulas (2) and (3) can express the reverse gradient of the loss function with respect to the input data and weights as:

[0052]

[0053] Or using separate padding, stride, and dilation parameters, expressed as:

[0054]

[0055] Where head is a tensor of the gradient of the loss with respect to the output

[0056] Strided(Head) is created by inserting s between each column of the head h –1 Zero column and insert s between each row of the header w –10 lines to implement Rot 180 (W) is the input weight tensor flipped 180° (or ).

[0057] For 4D data layout, where X has X(N,C in ,H,W) shape, W has W(C out ,C in ,k h ,k w ), the forward convolution is given by the following formula:

[0058]

[0059] Therefore, the two reverse gradients are:

[0060]

[0061] Among them, FlipRot(W): flip the tensor W in the H and W dimensions, and transpose it in the N and C dimensions.

[0062] Stretched(Head): Stretches the head tensor only in the above H and W dimensions,

[0063] T NC (X) : Transposes the N and C dimensions of the tensor x.

[0064] Using the expansion parameter, the gradient of the weights can be further simplified:

[0065] Conv(T NC (X),T NC(Strided(Head)))=Conv(T NC (X),T NC (Head), dilation = s forward )

[0066] In the case where stride > 1 and dilation > 1, the final formula becomes:

[0067]

[0068] For tensor processors with fractal data formats, convolution requires that the input data tensor must be in 5D NC1HWC0 format and the input weight tensor must be in fractal (C1HW)N1N0C0 format, where

[0069]

[0070] Where ' / / ' is integer division and '%' is integer modulo.

[0071] Figure 1 A simplified system for performing a reverse convolution operation 104 for use in training a CNN system is shown. In this diagram, the reverse convolution has three inputs: a head 118 is the gradient of the error (or loss) with respect to the output y 116 of the forward convolution 106 used in the forward pass, the data X 112, and the weights W 114. The reverse convolution must generate two gradients, one with respect to the data input X 120 (expressed as dL / dX) and the other with respect to the weight input W 122 (expressed as dL / dX). In addition, some tensor processors with fractal data formats require the data and output to be in 5D format, with the weights in fractal data format.

[0072] The embodiment of the present invention uses two convolution operators 108, 110 to calculate the gradients 120, 122. At the same time, the data format requirement requires the implementation of an appropriate data format converter. Figure 2 The system provided by the embodiment is shown.

[0073] Embodiments of the present invention may implement operations optimized for specific computer architectures, implementations, or input and output data formats. For example, some embodiments may be optimized for tensor processors and fractal data layouts.

[0074] The optimization process begins by determining whether the operator is a 2D convolution operator by examining its formula and properties. Then, two new Conv2D operators can be constructed for the input data and the reverse gradient of the input weights.

[0075] Examples of operators used in some embodiments include the following four supporting operators: FlipRot(X), Strided(X), TNC (X) and like Figure 2 shown.

[0076] Embodiments may also use intermediate or temporary data formats. Data format conversion may first be represented as a series of multiple format conversions, which are then combined into a final composite transformation. In some embodiments, temporary data formats may be used, such as 4D format, NCHW, and 6D format, N1N0C1C0HW.

[0077] In an embodiment, data transformations may be expressed using tensor iterators as follows:

[0078] If X.shape = (H, W), Y.shape = (H, W) and X ij =Y ij ,but

[0079] i X =i Y ;j X =j Y

[0080] Flip tensor X ij =Y H–1–i,W–1–j The example can be written as:

[0081] i X =H–1–i Y ;j X =W–1–j Y

[0082] Using these notations, a specific data converter can be identified as follows.

[0083] For the transformation Strided(X), both the input tensor and the result need to be in 5D format. The 5D format of X with the corresponding iterator is:

[0084]

[0085] X with corresponding iterator strided The 5D format is:

[0086]

[0087] The relationship between the tensors using their respective iterators is:

[0088]

[0089] The corresponding code can be generated in a domain-specific computer language (such as TVM):

[0090]

[0091] For the transformation FlipRot(X), the tensor X is in fractal format and the transformation can be determined using chained interim steps. The first data in fractal format is transformed into NCHW format. Subsequently, the flip transformation is applied to convert the NCHW formatted data into fractal format. Given the original tensor X in NCHW format, the shape of the original tensor and the corresponding iterator can be expressed as:

[0092] (N,C,H,W)→[i0,i1,i2,i3]

[0093] Convert X to the shape The temporary 6D tiled tensor. The iterator of the temporary 6D tiled tensor can be represented as [j0,j1,j2,j3,j4,j5] and

[0094]

[0095] Convert to fractal format (Using the corresponding iterator) [l0, l1, l2, l3] we get:

[0096]

[0097] Applies a flip transformation to a tensor in NCHW format

[0098]

[0099] After flipping, the result can be transformed into fractal format Finally, in order to transform the formula of FlipRot(X) from fractal format to fractal format, these two transformations are linked so that Rewritten as a function of [l0,l1,l2,l3] as follows:

[0100]

[0101] in ,

[0102] In this case, due to symmetry, [l0,l1,l2,l3] can be rewritten as The formula for the function is analogous.

[0103] The corresponding code expressed in TVM is:

[0104]

[0105] For the transformation T NC(X), the input tensor X is in 5D format. Tensor X can be represented in NCHW format with corresponding iterators:

[0106] (N,C,H,W)→[i0,i1,i2,i3]

[0107] The 5D format of X with corresponding iterators is:

[0108]

[0109] as well as

[0110]

[0111] The N and C dimensions of X can be transposed by exchanging the corresponding iterators:

[0112] (C,N,H,W)→[i1,i0,i2,i3]

[0113] Then, the result of 5D format is in:

[0114]

[0115] T NC (X) The transformation from 5D format to 5D format can be determined by writing [j0, j1, j2, j3, j4] as a function of [k0, k1, k2, k3, k4]:

[0116]

[0117] Due to symmetry, the formula for writing [k0,k1,k2,k3,k4] as a function of [j0,j1,j2,j3,j4] is analogous.

[0118] The corresponding implementation expressed in TVM is:

[0119]

[0120] The transformation T can be determined similarly NC Frac(X) to get a formula for transposing N and C in the following formula:

[0121]

[0122] Simultaneous conversion to fractal format

[0123]

[0124] The formula is as follows:

[0125]

[0126] The corresponding implementation expressed in TVM is:

[0127]

[0128]

[0129] Figure 3 An embodiment of using data transformation to implement a reverse convolution pass when training a CNN is shown. The gradient of the head with respect to the data 120 is obtained by convolving 108 the weights 114 transformed by the FlipRot transform 202 and the head 118 transformed by the Strided transform 204. The intermediate values ​​of the gradient of the head with respect to the weights 122 are obtained by convolving 108 the weights 114 transformed by the FlipRot transform 202 and the head 118 transformed by the Strided transform 204. NC Transformation 208 transforms the data 112 and passes T NC The head 118 of the Frac transform 206 is convolved 110. In this embodiment, the intermediate gradient is then obtained by T NC • Frac transform 206 transform to produce the gradient of the head with respect to the weights 122 .

[0130] Embodiments of the present invention may be used in other applications and other frameworks where the reverse convolution may be expressed as a forward convolution.

[0131] Embodiments may use an iterative approach to perform data conversion and transformation using the methods herein.

[0132] Figure 4 An exemplary hardware implementation of a computing system 400 provided in an embodiment is shown. The computing system 400 can be a system-on-chip (SoC), a card, a module, a coprocessor, and can be located inside or outside the host system. The system 400 includes a task scheduler that monitors and controls the use of other system assets. The task scheduler can be one or more traditional CPUs that can have the same or different architectures. In some embodiments, it can also be a simplified controller or state machine. The task scheduler is coupled to an interconnect 404 that can access other blocks of the system. The network I / F 406 is used to send and receive data to / from external devices or host devices, and can include wired or wireless network technologies such as Ethernet. The network I / F 406 can also include parallel or serial bus technologies such as PCI, Firewire, Thunderbolt, SATA, etc. The I / O 408 is similar to the network I / F, but includes interfaces to connect to peripheral devices, user interfaces, low-speed interfaces such as low-speed USB or I 2C. Firmware 410 includes non-volatile memory that stores firmware, BIOS, boot code, configuration parameters, etc. used when the system is powered on, reset, or restarted. Memory / buffer 412 includes internal volatile and non-volatile memory required by system 400. DMA 414 can be used to read, write, and transfer blocks of memory within and external to system 400 with minimal input required by task scheduler 402. Memory I / F 416 is used to communicate with external memory (e.g., SRAM, DRAM, flash memory, etc.).

[0133] Core 420 includes a dedicated convolution functional unit to accelerate convolution operations. In some embodiments, there are multiple cores. In some embodiments, the core will also include hardware for performing data format conversion and im2col operations. Interface 422 provides an interface for the task scheduler or DMA 414 to receive data or instructions for core 420. DMA engine 424 can be used to read or write data from network I / F 406, memory / buffer 412, or memory I / F 416 to core 420. Before data from an external source can be used, it may need to be transformed using transform 428. The transformed data is copied into two buffers A 430 and B 432 before being loaded into input registers 434. In some applications, the contents of buffers A and B will be the data and weights for the convolution operation. Matrix multiplication unit 436 and accumulation unit 438 are used to perform convolution as needed. The results are stored in output registers 440 for use outside of core 420. This exemplary core 420 has variations and can be customized and optimized for specific applications and operations.

[0134] Convolution function unit 420 typically has restrictions on the size of input data. The minimum input data (called a fractal) is typically a square matrix of a predetermined length (e.g., 16). Input data to convolution function unit 420 should be a multiplier of the fractal. The hardware also provides fractal instructions, which are more efficient than scalar and vector instructions. These factors are taken into account during convolution code generation.

[0135] To represent such a matrix, a 4D format is used, where two dimensions represent the index of the fractal (called the block index) and the other two dimensions represent the index of the elements within the fractal (called the fractal index). Figure 5 In

[15] , matrix A502 is composed of four small fractals Aij, where 1≤i≤2 and 1≤j≤2. Each fractal Aij is a 16×16 matrix {Aijkl}, where 1≤k and l≤16. Here, i and j are the indices of the fractal, and k and l are the indices of the elements within the fractal. To transpose the matrix in fractal format, you need to swap {i,j} (the position of the transposed fractal) and {k,l} (the transposed fractal).

[0136] 1 For i, j, k, l:

[0137] 2A_transpose[i,j,k,l]=A[j,i,l,k]

[0138] In an embodiment, the convolution function 420 treats the input matrix 502 as a block matrix, and each block is a fractal. Convolution is performed by block matrix multiplication, such as Figure 5 shown.

[0139] Figure 6 An embodiment of a general purpose computing platform is shown that can perform the functions of a host processor in a larger system, or interface with a co-processor system, ASIC, card, or module that accelerates convolution operations. Figure 4 6 is a block diagram of a computing system that can be used to implement the various devices and methods disclosed herein. A particular device may use all of the components shown or only a subset of these components, and the degree of integration between devices may vary. In addition, a device may contain multiple instances of a component, such as multiple processing units, processors, memories, transmitters, receivers, etc. The computing system includes a processing unit 600. The processing unit 600 typically includes a central processing unit (CPU) 610, a bus and memory 620, and may optionally also include a mass storage device 630, a video adapter 640, and an I / O interface 660 (each device is shown in dashed lines to indicate that they are optional). The computing system may also include one or more network interfaces 650 for connecting the computing system to a communication network 655.

[0140] The CPU may include any type of electronic data processor and may include one or more cores or processing elements. The memory may include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or a combination thereof. In one embodiment, the memory may include a ROM used at startup and a DRAM used to store programs and data when executing programs. The bus may be one or more of several bus architectures of any type, including a memory bus or a memory controller bus, a peripheral bus, or a video bus.

[0141] Mass storage may include any type of non-transitory storage device for storing data, programs, and other information and making them accessible via a bus. Mass storage may include, for example, one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.

[0142] Video adapter 640 and I / O interface 660 provide optional interfaces to couple external input and output devices to the processing unit. Examples of input and output devices include a display 670 coupled to video adapter 640 and an I / O device 580, such as a touch screen, coupled to I / O interface 660. Other devices can be coupled to the processing unit, and other or fewer interfaces can be used. For example, a serial interface (not shown) such as a universal serial bus (USB) can be used to provide an external device interface. Alternatively, the computing system can rely on one or more network interfaces to connect to available mass storage, video adapters, and available I / O interfaces on the network.

[0143] According to an embodiment of the present invention, a system for accelerating convolution operations is provided, comprising: a task scheduler; a plurality of cores, each of the plurality of cores including a convolution functional unit, each of the convolution functional units configured to receive first data in a first format, receive second data in a second format, and output third data in a third format. The system further comprises: an I / O interface coupled to the task scheduler; and a non-transitory computer-readable storage medium storing software instructions for controlling the task scheduler. The task scheduler performs the steps of receiving a first data set and a second data set from the I / O interface, transforming the first data set into a first transformed data set having the first format, transforming the second data set into a second transformed data set having the second format, loading the first transformed data set and the second transformed data set into the convolution functional unit, and receiving a result in the third format from the convolution functional unit.

[0144] In another embodiment, the software instructions further control the task scheduler to execute the step of transforming the result into a conversion result.

[0145] In another embodiment, the result is in a 5D format and the converted result is in a fractal format.

[0146] In another embodiment, the first data set is a data set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

[0147] In another embodiment, the first data set is in 5D format, the first transformed data set is in 5D format, the second data set is in 5D format, and the second transformed data set is in fractal format.

[0148] In further embodiments, the result is a gradient with respect to the set of weights.

[0149] In another embodiment, the first data set is a set of weights of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

[0150] In another embodiment, the first data set is in a fractal format, the first transformed data set is in a fractal format, the second data set is in a 5D format, and the second transformed data set is in a 5D format.

[0151] In further embodiments, the result is a gradient relative to the data set.

[0152] In further embodiments, said gradient relative to said dataset is in a 5D format.

[0153] In some embodiments, the system further comprises: a retargetable data converter, wherein said transforming said first data set into a first transformed data set is performed by a vector or scalar functional unit on a core running said retargetable data converter to perform said transform.

[0154] In another embodiment, the convolution function processes a three-dimensional matrix.

[0155] In another embodiment, the convolution functional unit performs an img2col function followed by a matrix multiplication function.

[0156] In further embodiments, the first data set and the second data set are in tensor format.

[0157] According to another aspect, an embodiment includes a method for accelerating a convolution operation. The method includes: a task scheduler receiving a first data set and a second data set from an I / O interface. The task scheduler transforms the first data set into a first transformed data set, the first transformed data set having a first format. The task scheduler transforms the second data set into a second transformed data set, the second transformed data set having a second format. The task scheduler loads the first transformed data set and the second transformed data set into a convolution functional unit, the convolution functional unit being configured to receive first data in a first format, receive second data in a second format, and output third data in a third format. The task scheduler receives a result in the third format from the convolution functional unit.

[0158] Another aspect of the present invention includes a method for transforming a tensor. The method includes receiving an input tensor having an input format having an input iterator set, and storing the input tensor in an input memory. Allocating output memory to store an output tensor having an output format having an output iterator set. Iterating over the output memory to copy input components of the input tensor to a portion of the output memory corresponding to output components of the output tensor, the input components having indices defined as functions of the output iterator set.

[0159] In some embodiments, the method further includes: allocating a chain memory to store a chain tensor having a chain format, the chain format having a chain iterator set; iterating on the chain memory to copy the output component of the output tensor to a portion of the chain memory corresponding to the chain component of the chain tensor, the index of the output component being defined as a second function of the chain iterator set.

[0160] Additional embodiments include: for the output tensor Y[i], the output format has the form (HW, 1), the output iterator set is i, for the input tensor X[j], the input format has the form (H, W), the input iterator set is j0, j1, and the function is Y[i] = X[i / W, i%W].

[0161] Additional embodiments include: for an output tensor Y[i], the output format has the form (CHW, 1), the output iterator set is i, for an input tensor X[j], the input format has the form (C, H, W), the input iterator set is j0, j1, j2, the function is

[0162] Additional embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has the form (HW, 1), the input iterator set is j0, and the function is Y[i0, i1] = X[i0*W+i1].

[0163] Additional embodiments include: for an output tensor Y[i], the output format has the form (C, H, W), the output iterator set is i0, i1, i2, for an input tensor X[j], the input format has the form (CHW, 1), the input iterator set is j0, the function is Y[i0, i 1, i2]=X[i0*HW+i1*W+i2].

[0164] Further embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has the form (H, W), the input iterator set is j0, j1, and the function is Y[i0, i1] = X[H–1–i0, W–1–i1].

[0165] Additional embodiments include: for the output tensor Y[i], the output format has the form (H, W), the output iterator set is i0, i1, for the input tensor X[j], the input format has (H, W), the input iterator set is j0, j1, and the function is Y[i] = X[i1, i0].

[0166] Unless defined otherwise, all technical and / or scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art.

[0167] Through the description of the above embodiments, the present invention can be implemented only by hardware, or by software and necessary general hardware platforms. Based on such understanding, the technical solution of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile or non-transient storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash drive, or a removable hard disk. The software product includes many instructions that enable a computer device (a personal computer, a server, or a network device) to execute the method provided in the embodiment of the present invention. For example, such execution can correspond to the simulation of the logical operation described herein. According to an exemplary embodiment, the software product can additionally or alternatively include multiple instructions that enable a computer device to perform the operation of configuring or programming a digital logic device.

[0168] Although the present invention has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations are possible without departing from the scope of the invention. The specification and drawings are therefore to be considered merely as illustrative of the invention as defined by the appended claims and any and all modifications, variations, combinations or equivalents falling within the scope of this specification are contemplated.

Claims

1. A system for accelerating convolution operations, characterized in that: The system comprises: Task scheduler; a plurality of cores, each core of the plurality of cores comprising a convolutional functional unit, each of the convolutional functional units configured to receive first data in a first format, receive second data in a second format, and output third data in a third format; An I / O interface coupled to the task scheduler; A non-transitory computer-readable storage medium for storing software instructions, wherein the software instructions control the task scheduler to perform the following operations: receiving a first data set and a second data set from the I / O interface; transforming the first data set into a first transformed data set, the first transformed data set having the first format; transforming the second data set into a second transformed data set, the second transformed data set having the second format; The first data set is in 5D format, the first transformed data set is in 5D format, the second data set is in 5D format, and the second transformed data set is in fractal format; or the first data set is in fractal format, the first transformed data set is in fractal format, the second data set is in 5D format, and the second transformed data set is in 5D format; loading the first transformed data set and the second transformed data set into the convolution function unit; A result of the convolution operation in the third format is received from the convolution functional unit.

2. The system according to claim 1, wherein: The software instructions further control the task scheduler to transform the result into a conversion result.

3. The system according to claim 2, characterized in that The result is in 5D format, and the converted result is in fractal format.

4. The system according to any one of claims 1 to 3, characterized in that The first data set is a data set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

5. The system according to claim 4, characterized in that The result is the gradient with respect to the set of weights.

6. The system according to claim 1, wherein: The first data set is a weight set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

7. The system according to claim 1 or 6, characterized in that The result is the gradient with respect to the dataset.

8. The system according to claim 7, characterized in that The gradient relative to the data set is in 5D format.

9. The system according to any one of claims 1 to 3 and 6, characterized in that Also included is a retargetable data converter, wherein said transforming said first data set into a first transformed data set is performed by a vector or scalar functional unit on a core executing said retargetable data converter to perform said transform.

10. The system according to any one of claims 1 to 3 and 6, characterized in that The convolution function unit processes a three-dimensional matrix.

11. The system according to any one of claims 1 to 3 and 6, characterized in that: The convolution function unit performs the img2col function and then performs the matrix multiplication function.

12. The system according to claim 1, wherein: The first dataset and the second dataset are in tensor format.

13. A method for accelerating convolution operation, characterized in that: The method comprises: The task scheduler receives the first data set and the second data set from the I / O interface; The task scheduler transforms the first data set into a first transformed data set, wherein the first transformed data set has a first format; The task scheduler transforms the second data set into a second transformed data set, wherein the second transformed data set has a second format; The first data set is in 5D format, the first transformed data set is in 5D format, the second data set is in 5D format, and the second transformed data set is in fractal format; or the first data set is in fractal format, the first transformed data set is in fractal format, the second data set is in 5D format, and the second transformed data set is in 5D format; The task scheduler loads the first transformed data set and the second transformed data set into a convolution function unit, the convolution function unit being configured to receive first data in a first format, receive second data in a second format, and output third data in a third format; The task scheduler receives the result of the convolution operation in the third format from the convolution functional unit.

14. The method according to claim 13, characterized in that Also includes: The task scheduler transforms the result into a conversion result.

15. The method according to claim 14, characterized in that The result is in 5D format, and the converted result is in fractal format.

16. The method according to any one of claims 13 to 15, characterized in that The first data set is a data set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

17. The method according to claim 16, characterized in that The result is the gradient with respect to the set of weights.

18. The method according to claim 13, characterized in that The first data set is a weight set of a second convolution operation, and the second data set is a propagated error of the second convolution operation of the data set and the weight set.

19. The method according to claim 13 or 18, characterized in that The result is the gradient with respect to the dataset.

20. The method according to claim 19, characterized in that The gradient relative to the data set is in 5D format.

21. The method according to any one of claims 13 to 15 and 18, characterized in that The transforming of the first data set into a first transformed data set is performed by a vector or scalar function unit on a core running a retargetable data converter to perform the transform.

22. The method according to any one of claims 13 to 15 and 18, characterized in that The convolution function unit processes a three-dimensional matrix.

23. The method according to any one of claims 13 to 15 and 18, characterized in that The convolution function unit performs the img2col function and then performs the matrix multiplication function.

24. The method according to claim 13, wherein The first dataset and the second dataset are in tensor format.

25. A method for transforming a tensor, characterized in that The method comprises: receiving an input tensor having an input format having an input iterator set, and storing the input tensor into an input memory; Allocating output memory to store output tensors having an output format having an output iterator set; iterating over the output memory to copy input components of the input tensor to a portion of the output memory corresponding to output components of the output tensor, the index of the input component being defined as a function of the output iterator set; Wherein, when the input format is a 5D format, the output format is a 5D format or a fractal format; when the input format is a fractal format, the output format is a fractal format.

26. The method according to claim 25, characterized in that The method further comprises: Allocate chained memory to store chained tensors in a chained format having a chained iterator set; Iterating over the chained memory to copy an output component of the output tensor to a portion of the chained memory corresponding to a chained component of the chained tensor, the output component having an index defined as a second function of the chained iterator set.

27. The method according to claim 25, characterized in that For the output tensor Y [ i ], the output format has the form ( HW , 1), the output iterator set is i , for the input tensor X [ j ], the input format has the form ( H , W ), the input iterator set is j 0. j 1, the function is Y [ i ] = X [ i / W , i % W ],in, .

28. The method according to claim 25, characterized in that For the output tensor Y [ i ], the output format has the form ( CHW , 1), the output iterator set is i , for the input tensor X [ j ], the input format has the form ( C , H , W ), the input iterator set is j 0. j 1. j 2. The function is Y [ i ] = X [ , , ],in, .

29. The method according to claim 25, characterized in that For the output tensor Y [ i ], the output format has the form ( H , W ), the output iterator set is i 0. i 1. For the input tensor X [ j ], the input format has the form ( HW ,1), the input iterator set is j 0, the function is Y [ i 0, i 1] = X [ ],in, .

30. The method according to claim 25, wherein For the output tensor Y [ i ], the output format has the form ( C , H , W ), the output iterator set is i 0. i 1. i 2. For the input tensor X [ j ], the input format has the form ( CHW , 1), the input iterator set is j 0, the function is Y [ i 0, i 1, i 2] = X [ ],in, .

31. The method according to claim 25, wherein For the output tensor Y [ i ], the output format has the form ( H , W ), the output iterator set is i 0. i 1. For the input tensor X [ j ], the input format has the form ( H , W ), the input iterator set is j 0. j 1, the function is Y [ i 0, i 1] = X [ , ],in, .

32. The method according to claim 25, wherein For the output tensor Y [ i ], the output format has the form ( H , W ), and the output iterator set is i 0. i 1. For the input tensor X [ j ], the input format has the form ( H , W ), the input iterator set is j 0. j 1, the function is Y [ i ] = X [ i 1, i 0], where .

33. A computer storage medium, characterized in that Used to store computer instructions, when one or more processors execute the instructions, the one or more processors execute the method according to any one of claims 13-24, or the method according to any one of claims 25-32.

Images