Tiled Format for Convolutional Neural Networks

Abstract

The present invention relates to a tiled format for a convolutional neural network. A system, an apparatus and a method for converting data into a tiled format when implementing a convolutional neural network are disclosed. A system includes at least a memory, a cache, a processor and a plurality of computing units. The memory stores a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data. The processor converts the first and second buffers from a linear format to a third and fourth buffer in a tiled format, respectively. The plurality of computing units load the tiled format data from the third and fourth buffers in the memory into a cache and then perform a convolution filter operation on the tiled formatted data. The system generates a classification of a first data set based on the result of the convolution filter operation.

Description

Technical Field

[0001] The present invention relates generally to the field of computers, and more particularly to a tiled format for convolutional neural networks. Background Art

[0002] Performing convolution filter operations in a parallel processing unit involves loading large amounts of data from memory, which results in high memory bandwidth utilization. Loading filters and input data elements typically requires calculating offsets in order to step through data elements stored in memory in a linear format. However, this requires complex formulas for each load instruction to calculate the offsets used to move through the elements of the data set in the correct order to perform the convolution filter operation. As used herein, the term "linear format" is defined as a format in which consecutive data set elements are stored in adjacent memory locations in a sequential manner until the end of a physical row or column of the memory is reached. Examples of linear formats for storing data set elements include row-major and column-major. In a row-major format, consecutive elements of a data set row are adjacent to each other in memory. In a column-major format, consecutive elements of a data set column are adjacent to each other in memory. Processing data set elements in a linear format results in inefficient use of memory bandwidth and computing unit processing resources. Summary of the invention

[0003] Some aspects of the invention can be described as follows:

[0004] 1. A system, which may include:

[0005] a memory storing a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data;

[0006] Cache;

[0007] A processor configured to:

[0008] converting the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer; and

[0009] converting the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer; and

[0010] A plurality of computing units, wherein the plurality of computing units are configured to:

[0011] loading data elements of the third buffer and the fourth buffer from the memory into the cache; and

[0012] performing a convolution filter operation on the data elements of the third buffer and the fourth buffer to generate a result in the tiled format;

[0013] Therein, the system is configured to implement classification of the first data set based on the results from the convolution filter operation.

[0014] 2. A system according to clause 1, wherein the cache comprises a plurality of channels, wherein the plurality of computational units are configured to load convolution filter data elements from the third buffer in parallel on the plurality of channels within a single clock cycle.

[0015] 3. The system of clause 1, wherein the cache comprises a plurality of channels, wherein the plurality of computational units are configured to load image data elements from the fourth buffer in parallel on the plurality of channels within a single clock cycle.

[0016] 4. The system of clause 1, wherein converting the first buffer from the linear format to the third buffer in the tiled format comprises:

[0017] reading values ​​from sequential locations of said first buffer in said memory; and

[0018] Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

[0019] 5. The system of clause 1, wherein converting the second buffer from the linear format to the fourth buffer in the tiled format comprises:

[0020] reading values ​​from sequential locations of said second buffer in said memory; and

[0021] Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

[0022] 6. A system according to clause 1, wherein the convolution filter data includes a plurality of convolution filters, wherein each convolution filter in the plurality of convolution filters has three rows and three columns.

[0023] 7. A system according to clause 1, wherein the classification of the first data set is implemented during execution of a machine learning engine application.

[0024] 8. A method, which may include:

[0025] storing, by a memory, a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data;

[0026] converting, by a processor, the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer;

[0027] converting, by a processor, the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer;

[0028] loading data elements of the third buffer and the fourth buffer from the memory into a cache through a plurality of computation units;

[0029] performing convolution filter operations on the data elements of the third buffer and the fourth buffer by a plurality of computing units to generate results in the tiled format; and

[0030] Classification of the first data set is achieved based on the results from the convolutional filter operation.

[0031] 9. A method according to clause 8, wherein the cache includes multiple channels, and wherein the method further includes loading convolution filter data from the third buffer on the multiple channels in parallel within a single clock cycle through the multiple computing units.

[0032] 10. The method of clause 8, wherein the cache comprises a plurality of channels, and wherein the method further comprises loading the image data from the fourth buffer in parallel on the plurality of channels within a single clock cycle by the plurality of computing units.

[0033] 11. The method of clause 8, wherein converting the first buffer from the linear format to the third buffer in the tiled format comprises:

[0034] reading values ​​from sequential locations of said first buffer in said memory; and

[0035] Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

[0036] 12. The method of clause 8, wherein converting the second buffer from the linear format to the fourth buffer in the tiled format comprises:

[0037] reading values ​​from sequential locations of said second buffer in said memory; and

[0038] Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

[0039] 13. A method according to clause 8, wherein the convolution filter data comprises a plurality of convolution filters, wherein each convolution filter in the plurality of convolution filters has three rows and three columns.

[0040] 14. The method of clause 8, further comprising implementing the classification of the first data set during execution of a machine learning engine application.

[0041] 15. An apparatus, which may include:

[0042] a memory storing a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data; and

[0043] A processor configured to:

[0044] converting the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer; and

[0045] converting the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer; and

[0046] Classification of the first data set is achieved based on results from convolution filter operations performed on data elements stored in the third buffer and the fourth buffer in a tiled format.

[0047] 16. An apparatus according to clause 15, further comprising a cache, wherein the cache comprises a plurality of channels, and wherein the apparatus is further configured to load convolution filter data from the third buffer in parallel on the plurality of channels within a single clock cycle.

[0048] 17. The apparatus of clause 15, further comprising a cache, wherein the cache comprises a plurality of channels, and wherein the apparatus is further configured to load the image data from the fourth buffer in parallel on the plurality of channels within a single clock cycle.

[0049] 18. The apparatus of clause 15, wherein converting the first buffer from the linear format to the third buffer in the tiled format comprises:

[0050] reading values ​​from sequential locations of said first buffer in said memory; and

[0051] Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

[0052] 19. The apparatus of clause 15, wherein converting the second buffer from the linear format to the fourth buffer in the tiled format comprises:

[0053] reading values ​​from sequential locations of said second buffer in said memory; and

[0054] Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

[0055] 20. The apparatus of clause 15, further comprising:

[0056] The classification of the first data set is achieved during the process. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0058] Figure 1 It is a block diagram of one implementation of a computing system.

[0059] Figure 2 is a block diagram of another implementation of a computing system.

[0060] Figure 3 It is a block diagram of one implementation of a matrix operation unit.

[0061] Figure 4 is a schematic diagram of one implementation of the data layout of the first source matrix operated by the SIMD unit.

[0062] Figure 5 is a schematic diagram of one implementation of the data layout of the second source matrix operated by the SIMD unit.

[0063] Figure 6 This is a schematic diagram of one implementation of a 32K2C1R1S convolution filter data block.

[0064] Figure 7 This is a schematic diagram of one implementation of a 32K8C convolution filter data block.

[0065] Figure 8 is a schematic diagram of one implementation of a convolution filter data block in a tiled format.

[0066] Fig. 9 It is a schematic diagram of one implementation method of the image data block.

[0067] Fig.10 It is a schematic diagram of one implementation of the 4C18H34W image data block.

[0068] Fig.11 is a schematic diagram of one implementation of an image stored in a memory.

[0069] Fig.12 An example of pseudo code for converting convolution filter data from a linear format to a tiled format is shown according to one implementation.

[0070] Fig.13 An example of pseudo code for converting image data from a linear format to a tiled format is shown according to one implementation.

[0071] Fig.14 is a generalized flow chart illustrating one implementation of a method for implementing a convolutional layer of a machine learning engine using a tiled format.

[0072] Fig.15 is a generalized flow chart illustrating one implementation of a method for transforming a first buffer of convolution filter data from a linear format to a second buffer in a tiled format.

[0073] Fig.16 is a generalized flow chart illustrating one implementation of a method for transforming a first buffer of image data from a linear format to a second buffer in a tiled format. DETAILED DESCRIPTION

[0074] In the following description, many specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those of ordinary skill in the art will recognize that various implementations can be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid making the methods described herein difficult to understand. It should be understood that, for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the size of some elements may be enlarged relative to other elements.

[0075] Various systems, devices and methods for converting data into a tiled format to implement a convolutional neural network are disclosed herein. A system includes at least a memory, a cache, a processor and a plurality of computing units. The memory stores a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data. The processor converts the convolution filter data in the first buffer from a linear format to a tiled format and then stores the convolution filter data in the tiled format to a third buffer. The processor also converts the image data in the second buffer from a linear format to a tiled format and then stores the image data in the tiled format to a fourth buffer.

[0076] In one implementation, the term "tile format" or "tiled format" is defined as a format for storing data elements in memory locations such that data elements used to perform a convolution filter operation are stored in adjacent memory locations. In one implementation, the term "convolution filter operation" is defined as an operation performed as part of a convolution layer that implements a neural network. In another implementation, the term "tile format" is defined as a format in which data elements that make up a column of a tile are stored in adjacent memory locations. The tiled format causes the data elements to be ordered in a particular layout so that the data elements can be loaded contiguously without performing offset calculations. The term "tile" is defined as an N×M block of elements, where N and M are positive integers, and where at least one of N or M is greater than 1. A "tile" may also be referred to as a "block" herein.

[0077] In one implementation, multiple computing units load a third buffer and a fourth buffer from a memory to a cache, and then perform convolution filter operations on the data elements of the third and fourth buffers to generate results in a tiled format. The system generates a classification of the first data set based on the tiled results generated by the convolution filter operations. For example, in one implementation, the system performs multiple convolution filter operations while implementing a neural network to classify an image into one or more categories. The neural network can be a convolutional neural network, a recursive neural network, or other types. Various tasks such as handwritten digit classification and facial detection can be performed by the neural network. In addition, the neural network can perform other more challenging visual classification tasks. Other applications of neural networks include speech recognition, language modeling, sentiment analysis, text prediction, etc. In other implementations, the system performs multiple convolution filter operations on multiple computing units that are part of other types of software applications.

[0078] In one implementation, converting the first buffer from a linear format to a third buffer in a tiled format involves a first stage of reading values ​​from sequential locations of the first buffer in memory, wherein the first buffer stores data elements in the linear format. Then, converting the first buffer from a linear format to a third buffer in a tiled format involves a second stage of stepping through the third buffer, wherein a stride is equal to a sum of the plurality of input channels and the plurality of convolution filters, while writing values ​​to the third buffer, wherein the third buffer stores data elements in the tiled format.

[0079] In one implementation, converting the second buffer from a linear format to a fourth buffer in a tiled format involves a first stage of reading values ​​from sequential locations of the second buffer in memory, wherein the second buffer stores elements in a linear format. Then, converting the second buffer from a linear format to a fourth buffer in a tiled format involves a second stage of stepping through the fourth buffer, wherein the stride is equal to a plurality of pixel channels, while simultaneously writing values ​​to the fourth buffer, wherein the fourth buffer stores elements in a tiled format.

[0080] In one implementation, storing data elements in a tiled format increases cache hit efficiency. In a typical application, each computing unit processes a different portion of the data elements, but these portions can be mapped to the same cache channel. This affects cache efficiency because different computing units will eventually request data via the same cache channel. Therefore, the computing units will wait for data to be returned from the cache, and the cache will process requests one by one in the same channel. However, when data elements are stored in a tiled format, different computing units are mapped to different channels. When computing units execute the same instruction set in parallel, requests will be sent to the cache through different channels, which helps to improve cache access efficiency.

[0081] Reference now Figure 1 , a block diagram of one implementation of a computing system 100 is shown. In one implementation, computing system 100 includes at least processors 105A-N, input / output (I / O) interface 120, bus 125, memory controller 130, network interface 135, memory device 140, display controller 150, and display 155. In other implementations, computing system 100 includes other components and / or computing system 100 is arranged in a different manner. Processors 105A-N represent any number of processors included in system 100.

[0082] In one implementation, processor 105A is a general purpose processor, such as a central processing unit (CPU). In one implementation, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. In some implementations, processors 105A-N include multiple data parallel processors. In one implementation, processor 105N is a GPU that provides pixels to display controller 150 to be driven to display 155.

[0083] The memory controller 130 represents any number and type of memory controllers accessible by the processors 105A-N. The memory controller 130 is coupled to any number and type of memory devices 140. The memory devices 140 represent any number and type of memory devices. For example, the memory types in the memory devices 140 include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc.

[0084] The I / O interface 120 represents any number and type of I / O interfaces (e.g., a peripheral component interconnect (PCI) bus, a PCI-Extension (PCI-X), a PCIE (PCI Express) bus, a Gigabit Ethernet (GBE) bus, a Universal Serial Bus (USB)). Various types of peripheral devices (not shown) are coupled to the I / O interface 120. These peripheral devices include (but are not limited to) a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, etc. The network interface 135 is used to receive and send network messages over a network.

[0085] In various implementations, computing system 100 is a computer, a laptop, a mobile device, a game console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or devices. Note that the number of components of computing system 100 varies from implementation to implementation. For example, in other implementations, the number of each component is less than 100. Figure 1 It should also be noted that in other implementations, the computing system 100 includes Figure 1 Other components not shown in FIG. In addition, in other implementations, the computing system 100 is used with Figure 1 The method shown in the figure is different from other methods of construction.

[0086] Now turn to Figure 2 , a block diagram of another implementation of a computing system 200 is shown. In one implementation, system 200 includes a GPU 205, a system memory 225, and a local memory 230. System 200 also includes other components that are not shown to avoid obscuring the figure. GPU 205 includes at least a command processor 235, a control logic 240, a scheduling unit 250, a computing unit 255A-N, a memory controller 220, a global data sharing 270, a level 1 (L1) cache 265, and a level 2 (L2) cache 260. In other embodiments, GPU 205 includes other components, omits one or more of the components shown, has multiple instances of a component, although Figure 2 Only one example is shown, and / or may be organized in other suitable ways. In one implementation, the circuitry of GPU 205 includes ( Figure 1 ) processor 105N.

[0087] In various implementations, the computing system 200 executes any of a variety of types of software applications. As part of executing a given software application, a host CPU (not shown) of the computing system 200 launches a kernel to be executed on the GPU 205. The command processor 235 receives the kernel from the host CPU and uses the scheduling unit 250 to issue a corresponding wavefront to the computing units 255A-N. In one implementation, each computing unit 255A-N includes a matrix operation unit. For example, in one implementation, the matrix operation unit performs matrix multiplication operations to implement a convolutional layer of a neural network. Additionally, in various implementations, the matrix operation unit performs other types of matrix operations. The wavefronts executed on the computing units 255A-N read and write data to the global data share 270, the L1 cache 265, and the L2 cache 260 within the GPU 205. Although not shown in Figure 2 , but in one implementation, the computing units 255A-N also include one or more caches and / or local memories within each computing unit 255A-N.

[0088] In one implementation, the convolution filter data and the image data are stored in a linear format in the system memory 225 and / or the local memory 230. Before loading the convolution filter data and the image data into the L2 cache 260, the convolution filter data and the image data are converted from the linear format to a tiled format. In one implementation, the term "tile format" is defined as storing elements together into cells of tiles, each tile storing N×M blocks of matrix elements, where N and M are positive integers. The "tile format" causes consecutive tiles to be stored in memory in a sequential manner.

[0089] In one implementation, the command processor 235 converts the convolution filter data and the image data from a linear format to a tiled format. In another implementation, a host processor (e.g., Figure 1 The processor 105A of the processor 105A) converts the convolution filter data and the image data from a linear format to a tiled format. Then, during the execution of the wavefront on the computing unit 255A-N, the convolution filter data and the image data are loaded to and from the L2 cache 260 in an efficient manner because the data is stored in a tiled format. In one implementation, when the convolution filter data and the image data are stored in a tiled format, the matrix data elements are read into the computing unit 255A-N in parallel from the L2 cache 260 on multiple channels. This is a more efficient method than storing the data elements in a linear format, which would cause the data elements to be read from the L2 cache 260 on a single channel in a serial manner.

[0090] Reference now Figure 3 , a block diagram of one implementation of a matrix operation unit 300 is shown. In one implementation, each computation unit 255A-N includes circuitry of a matrix operation unit 300. In one implementation, the matrix operation unit 300 includes at least an architecture vector general purpose register (VGPR) file 305, a derivation unit 310, an accumulation VGPR file 315, a single instruction multiple data (SIMD) unit 320, and a SIMD unit 325. It should be understood that the matrix operation unit 300 may also include any number of other components that are not shown in order to avoid obscuring the figure. Additionally, in other implementations, the matrix operation unit 300 is organized in other suitable ways.

[0091] In one implementation, SIMD unit 320 is a floating point unit for performing various floating point operations, and SIMD unit 325 is a matrix unit for performing various matrix operations (e.g., dot product operations, matrix multiplication operations). In one implementation, each individual input connected to the architecture VGPR file 305 and the accumulation VGPR file 315 is shown to have 16 channels, each channel has 32 bits. In other implementations, the input has other numbers of channels with other bit widths. In one implementation, when the data elements are stored in a tiled format, the SIMD unit 325 operates on the data elements more efficiently. Therefore, in this implementation, the data elements are converted from a linear format to a tiled format before being read into the architecture VGPR file 305 and / or the accumulation VGPR file 315. This enables the operation to be performed by the SIMD unit 325 in a more efficient manner.

[0092] Now turn to Figure 4, shows a schematic diagram of an implementation of a data layout 400 of a first source matrix operated by a SIMD unit. In one implementation, the SIMD unit 325 ( Figure 3 )according to Figure 4 400 is organized for reading in a first source matrix to perform a matrix multiplication operation of multiplying the first source matrix by the second source matrix. It should be noted that a "source matrix" may also be referred to as a "data set" in this article. For example, in this implementation, each SIMD unit has 64 threads in the data layout 400. Each thread of the data layout 400 corresponds to a channel of the SIMD unit. In other implementations, the SIMD unit includes other numbers (e.g., 32, 128) of channels.

[0093] Depending on the size of the blocks being processed, a different number of blocks may be mapped according to data layout 400. For example, if the size of each block is 32×4, two blocks (Blk0 and Blk1) are mapped from the VGPR file to the SIMD unit. If the size of each block is 16×4, four blocks (Blk0, Blk1, Blk2, and Blk3) are mapped from the VGPR file to the SIMD unit. If the size of each block is 4×4, 16 blocks (Blk0, Blk1, Blk2, Blk3, Blk4, Blk5, Blk6, Blk7, Blk8, Blk9, Blk10, Blk11, Blk12, Blk13, Blk14, and Blk15) are mapped from the VGPR file to the SIMD unit.

[0094] Reference now Figure 5 , a schematic diagram showing an implementation of a data layout 500 for a second source matrix operated by a SIMD unit. In one implementation, a VGPR file and a SIMD unit (e.g., SIMD unit 325 ( Figure 3 )). For example, in one implementation, data layout 500 defines connections for loading a second source matrix so as to perform a matrix multiplication operation between a first source matrix and a second source matrix. In one implementation, data layout 500 is organized for 64 threads. In other implementations, data layout 500 may be organized for other numbers of threads. Each thread of data layout 500 corresponds to a channel of a SIMD unit. Depending on the size of the block being processed (e.g., 4×32, 4×16, 4×4), different numbers of blocks (e.g., 2, 4, 16) may be mapped according to data layout 500.

[0095] Now turn to Figure 6, a schematic diagram of an implementation of a 32K2C1R1S block 600 of convolution filter data is shown. In one implementation, the convolution filter data set is composed of multiple 32K2C1R1S blocks, each 32K2C1R1S block 600 contains 1 shared weight from 32 filters and 2 input channels. In one implementation, the shared weight is one of the weights of the 3×3 convolution filter. Figure 6 As shown, the convolution filter data is represented as K[x]C[y]R[z]S[w], which specifies that the data is in filter offset x, channel offset y, row offset z, and column offset w. As used herein, for 32K2C1R1S block 600, the number preceding the letter refers to the dimension of the variable referenced by the letter. Therefore, 32K2C1R1S block 600 includes a single shared weight for 32 separate convolution filters for 2 separate input channels. It should be noted that the terms "kernel" and "filter" are used interchangeably herein.

[0096] Reference now Figure 7 , a schematic diagram showing an implementation of a 32K8C block 700 for convolution filter data. In one implementation, the 32K2C1R1S block shown within the 32K8C block 700 is based on Figure 6 The 32K2C1R1S blocks 600 are organized in a 32K2C1R1S block 600. In other implementations, the 32K8C block 700 may be constructed with other sizes and / or arrangements of constituent blocks. In one implementation, all weights of the 3×3 convolution filters for 8 input channels are combined together to create the 32K8C block 700. For example, the first row of the 32K8C block 700 includes a 32K2C1R1S block for channels [0:1], followed by a 32K2C1R1S block for channels [4:5], followed by a 32K2C1R1S block for channels [2:3], and finally a 32K2C1R1S block for channels [6:7], wherein these 32K2C1R1S blocks include weights for the first row R=0 and the first column S=0 of the corresponding 3×3 convolution filters. The second row of 32K8C blocks 700 includes a 32K2C1R1S block for channel [0:1], followed by a 32K2C1R1S block for channel [4:5], followed by a 32K2C1R1S block for channel [2:3], and finally a 32K2C1R1S block for channel [6:7], wherein these 32K2C1R1S blocks include weights of the first row R=0 and the second column S=1 of the corresponding 3×3 convolutional filters. The remaining rows of 32K8C blocks 700 continue using the same pattern.

[0097] Reference now Figure 8, shows a schematic diagram of an implementation of a convolution filter data block 800 in a tiled format. In one implementation, the filter data block 800 is composed of 64 32K8C blocks, each of which is based on the 32K8C block 700 ( Figure 7 ) organization. In one implementation, the filter data block 800 includes 128 filters and 128 channels. In other implementations, the filter data block 800 may include other numbers of filters and channels.

[0098] In one implementation, the leftmost block in the first row of convolution filter data blocks 800 includes a 32K8C block for filters [0:31] and channels [0:7]. Moving to the right, the next block in the first row includes a 32K8C block for filters [0:31] and channels [8:15]. This pattern continues until the rightmost block in the first row includes a 32K8C block for filters [0:31] and channels [120:127]. The second row of filter blocks 800 begins with a 32K8C block for filters [32:63] and channels [0:7], followed by a 32K8C block for filters [32:63] and channels [8:15], and continues for subsequent channels until the rightmost 32K8C block in the second row corresponds to filters [32:63] and channels [120:127]. This pattern continues for the remaining rows of filter blocks 800, as shown in FIG. Figure 8 As shown. It should be understood that the dimensions of the convolution filter data block 800 represent a specific implementation. In other implementations, other sizes and / or arrangements of blocks may be combined together to form the convolution filter data block 800.

[0099] Reference now Fig. 9 , a schematic diagram of an implementation of an image data block 900 is shown. In one implementation, block 900 is referred to as a 4C1H1W block. The number-letter pair 4C refers to that block 900 has 4 channels, the number-letter pair 1H refers to that block 900 has a height of 1 pixel, and the number-letter pair 1W refers to that block 900 has a width of 1 pixel. In addition, the labels within each box of block 900 are represented in the format B[x]C[y]H[z]W[w]. As used herein, the label B[x]C[y]H[z]W[w] indicates that the data is in batch offset x, channel offset y, image row offset z, and image column offset w. Therefore, the different labels have the following definitions: B is the image batch number, C is the pixel channel, H is the image height, and W is the image width.

[0100] like Fig. 9As shown, 4C1H1W block 900 includes a B[0]C[0]H[0]W[0] pixel, followed by a B[0]C[1]H[0]W[0] pixel, followed by a B[0]C[2]H[0]W[0] pixel, and followed by a B[0]C[3]H[0]W[0] pixel. Thus, 4C1H1W block 900 contains a single pixel from each of four separate pixel channels. It should be understood that Fig. 9 The illustration of the block 900 shown in FIG. 1 is only one example of a block constructed in one particular implementation. In other implementations, blocks of other dimensions and / or other constituent blocks may be constructed.

[0101] Now turn to Fig.10 , a schematic diagram of an implementation of a 4C18H34W block 1000 is shown. As used herein, the term "4C18H34W block" refers to a block having 4 channels, 18 rows of pixels, and 34 columns of pixels. In other embodiments, blocks 1000 of other sizes other than a height of 18 and a width of 34 may also be constructed. In one implementation, each 4C18H34W block 1000 includes 17 rows and 34 columns of 4C1H1W blocks. In one implementation, according to the 4C1H1W block 900 ( Fig. 9 ) is organized as a 4C1H1W block that is part of a 4C18H34W block 1000. Fig.10 As shown, the first row of 4C18H34W blocks 1000 consists of 33 4C1H1W blocks, from pixel position H=0, W=0 to pixel position H=0, W=33. The second row of 4C18H34W blocks 1000 consists of 33 4C1H1W blocks, from pixel position H=1, W=0 to pixel position H=1, W=33. This pattern continues until the bottom row of 4C18H34W blocks 1000 has 33 4C1H1W blocks, from pixel position H=17, W=0 to pixel position H=17, W=33.

[0102] Reference now Fig.11 , a schematic diagram of one implementation of an image 1100 stored in a memory is shown. Image 1100 represents a tiled format for storing image data in a memory according to one implementation. Image 1100 is a 64-batch, 128-channel, 34×34 image composed of 4C18H34W blocks. In one implementation, the 4C18H34W blocks of image 1100 are arranged according to 4C18H34W blocks 1000 ( Fig.10 )structure.

[0103] The first block of the first row of image 1100 is a 4C18H34W block of channels [0:3], rows [0:17], columns [0:33] of batch 0 of image data. The second block of the first row is a 4C18H34W block of channels [4:7], rows [0:17], columns [0:33] of batch 0 of image data. This pattern continues until the last block of the first row is a 4C18H34W block of channels [124:127], rows [0:17], columns [0:33] of batch 0 of image data. The second row of image 1100 begins with a 4C18H34W block of channels [0:3], rows [16:33], columns [0:33] of batch 0 of image data. The next block of the second row of image 1100, moving to the right, is a 4C18H34W block of channels [4:7], rows [16:33], columns [0:33] of batch 0 of image data. This pattern continues until the last block of the second row is a 4C18H34W block of columns [0:33], rows [16:33], channels [124:127] of batch 0 of image data. The third row of image 1100 begins with a 4C18H34W block of columns [0:33], rows [0:17], channels [0:3] of batch 1 of image data. The next block of the third row of image 1100 moving to the right is a 4C18H34W block of columns [0:33], rows [0:17], channels [4:7] of batch 1 of image data. This pattern continues until the last block of the third row is a 4C18H34W block of columns [0:33], rows [0:17], channels [124:127] of batch 1 of image data.

[0104] This pattern of rows continues until the bottom row of image 1100. The bottom row of image 1100 has a 4C18H34W block of columns [0:33] rows [16:33] channels [0:3] of batch 63 of image data. The second block of the bottom row is a 4C18H34W block of columns [0:33] rows [16:33] channels [4:7] of batch 63 of image data. The pattern continues until the last block of the bottom row is a 4C18H34W block of columns [0:33] rows [16:33] channels [124:127] of batch 63 of image data.

[0105] Now turn to Fig.12, an example of pseudocode 1200 for converting convolution filter data from a linear format to a tiled format according to one implementation is shown. Pseudocode 1200 includes definitions of variables C, K, R, and S, which have specific values ​​for a particular implementation. For example, for 128 channels, C equals 128, for 128 filters (i.e., kernels), K equals 128, for 3 rows per filter, R equals 3, and for 3 columns per filter, S equals 3. In other implementations, the values ​​of these variables may vary depending on the number of channels, the number of filters, and the size of the filters. In addition, other variables used in pseudocode 1200 may also vary depending on the size of the tile, the number of channels and the bit width of the computational unit and / or matrix multiplication unit, the number of cache channels, the definition of different fields of the corresponding software instructions, and the like.

[0106] In one implementation, there are two buffers in memory for convolution filter data. The buffer input_filter[] stores the convolution filter data in memory in a linear format. The buffer tiling_filter[] stores the convolution filter data in memory in a tiled format after performing the conversion. After the convolution filter data in input_filter[] is converted from a linear format to a tiled format and stored in tiling_filter[], the code executed on the computational unit loads the convolution filter data from tiling_filter[] into a cache (e.g., an L2 cache).

[0107] Reference now Fig.13 , shows an example of pseudo code 1300 for converting image data from a linear format to a tiled format. In one implementation, pseudo code 1300 is used to convert image data from a linear format to a tiled format. Once the image data is converted from a linear format to a tiled format, the computing unit loads the image data from a memory into a cache in the tiled format.

[0108] Pseudocode 1300 includes definitions of variables H, W, C, K, Hp, Wp, and IMAGE_SIZE, which have specific values ​​for a specific implementation. For example, in one implementation, H is equal to 32 for 32 rows per tile, W is equal to 32 for 32 columns per tile, C is equal to 128 for 128 channels, K is equal to 128 for 128 filters (i.e., kernels), Hp is equal to 34 for 34 rows per image, Wp is equal to 34 for 34 columns per image, and IMAGE_SIZE is equal to 34. In other implementations, the values ​​of these variables may vary depending on the number of channels, the size of the image, the size of the tile, etc. In addition, other variables used in pseudocode 1300 may also vary depending on the size of the tile, the number of channels and the bit width of the computational unit and / or matrix multiplication unit, the number of cache channels, the definition of different fields of the corresponding software instructions, etc.

[0109] In one implementation, there are two buffers in memory for image data. The buffer input_image[] stores the image data in a linear format in memory. After the conversion has been performed, the buffer tiling_image[] stores the convolution filter data in memory in a tiled format. After the convolution filter data in input_image[] is converted from a linear format to a tiled format and stored in tiling_image[], the code executed on the compute unit will load the image data from tiling_image[] in a tiled format into a cache (e.g., an L2 cache). Then, a matrix multiplication operation is performed to convert the pseudo code 1200 ( Fig.12 )'s elements of the buffer tiling_filter[] are multiplied by the corresponding elements of the buffer tiling_image[].

[0110] Now turn to Fig.14 , shows an implementation of method 1400 for implementing a convolutional layer of a machine learning engine using a tiled format. For discussion purposes, in the steps of this implementation, Figure 15-16 The steps are shown in order. However, it should be noted that in various implementations of the described method, one or more of the described elements are performed simultaneously, in a different order than shown, or are omitted entirely. Other additional elements are also performed as needed. Any of the various systems or devices described herein is configured to implement method 1400.

[0111] A processor (e.g., a host CPU) detects a request to perform a convolution filter operation using data stored in a linear format in a first and second buffer in a memory (block 1405). In response to detecting the request, the processor converts the convolution filter elements stored in the linear format in the first buffer to a tiled format and stores the convolution filter elements in the tiled format in a third buffer in the memory (block 1410). In pseudo code 1200 ( Fig.12 An example of converting convolution filter elements from a linear format to a tiled format is described in .

[0112] In addition, in response to detecting the request, the processor converts the image data stored in the second buffer in a linear format into a tiled format and stores the image data in the tiled format into a fourth buffer in the memory (block 1415). Fig.13 ) describes an example of converting image data from a linear format to a tiled format. Then, multiple computing units load the third buffer and the fourth buffer from the memory to the cache (box 1420). In one implementation, the multiple computing units are part of a GPU and the cache is an L2 cache. Next, the multiple computing units perform a convolution filter operation by implementing multiple matrix multiplication operations between elements of the third and fourth buffers (box 1425). Then, the computing system uses the result of the convolution filter operation to complete a first action associated with a given software application (box 1430). In one implementation, the first action is a classification of a first data set, and the given software application is a machine learning application. In one implementation, the first data set is an image, and the classification identifies a given category to which the image belongs. In another implementation, the first data set is a video, and the classification assigns the video to a given category. In other implementations, the first data set includes other types of data. In other implementations, other types of actions associated with other types of software applications are performed. After box 1430, method 1400 ends.

[0113] Reference now Fig.15, shows one implementation of a method for transforming a first buffer of convolution filter data from a linear format to a second buffer in a tiled format. A processor detects a request to perform a convolution filter operation using convolution filter data stored in a linear format (box 1505). In response to detecting the request, the processor reads values ​​from sequential positions of a first buffer, where the first buffer stores elements in a linear format (box 1510). The processor then steps through the second buffer, where the stride is equal to the sum of a plurality of input channels and a plurality of convolution filters, while writing values ​​to a second buffer, where the second buffer stores elements in a tiled format (box 1515). In other words, the processor writes the values ​​(which it reads from the first buffer) to positions in the second buffer separated by a given stride, where the given stride is equal to the sum of a plurality of input channels and a plurality of convolution filters. After box 1515, method 1500 ends.

[0114] Now go to Fig.16 , shows one implementation of a method for converting a first buffer of image data from a linear format to a second buffer in a tiled format. A processor detects a request to perform a convolution filter operation on image data stored in a linear format (box 1605). In response to detecting the request, the processor reads values ​​from sequential locations of a first buffer, where the first buffer stores elements in a linear format (box 1610). The processor then steps through the second buffer, where the stride is equal to a number of pixel channels, while writing the values ​​to the second buffer, where the second buffer stores elements in a tiled format (box 1615). In other words, the processor writes the values ​​(which it read from the first buffer) to locations in the second buffer separated by a given stride, where the given stride is equal to a number of pixel channels. After box 1615, method 1600 ends.

[0115] In various implementations, the method and / or mechanism described herein are implemented using program instructions of a software application. For example, it is conceivable that program instructions that can be executed by a general or special processor are executed. In various implementations, such program instructions are represented by a high-level programming language. In other implementations, the program instructions are compiled from a high-level programming language into a binary, intermediate or other form. Optionally, program instructions describing hardware behavior or design are written. Such program instructions can be represented by a high-level programming language such as C. Optionally, a hardware design language (HDL) such as Verilog is used. In various implementations, program instructions are stored on any one of various non-temporary computer-readable storage media. The storage medium can be accessed by a computing system during use to provide program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

[0116] It should be emphasized that the above implementations are only non-limiting examples of implementations. For those skilled in the art, many changes and modifications will become apparent once the above disclosure is fully understood. It is intended that the following claims be interpreted as including all such changes and modifications.

Claims

1. A system comprising: a memory storing a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data; Cache; A processor configured to: converting the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer; and converting the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer; and A plurality of computing units, wherein the plurality of computing units are configured to: loading data elements of the third buffer and the fourth buffer from the memory into the cache; and performing a convolution filter operation on the data elements of the third buffer and the fourth buffer to generate a result in the tiled format; Therein, the system is configured to implement classification of the first data set based on the results from the convolution filter operation.

2. The system of claim 1 , wherein the cache comprises a plurality of channels, wherein the plurality of computational units are configured to load the convolution filter data elements from the third buffer in parallel on the plurality of channels within a single clock cycle.

3. The system of claim 1, wherein the cache comprises a plurality of channels, wherein the plurality of computational units are configured to load image data elements from the fourth buffer in parallel on the plurality of channels within a single clock cycle.

4. The system according to claim 1, wherein: Converting the first buffer from the linear format to the third buffer in the tiled format comprises: reading values ​​from sequential locations of said first buffer in said memory; and Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

5. The system according to claim 1, wherein: Converting the second buffer from the linear format to the fourth buffer in the tiled format comprises: reading values ​​from sequential locations of said second buffer in said memory; and Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

6. The system of claim 1, wherein the convolution filter data comprises a plurality of convolution filters, wherein each of the plurality of convolution filters has three rows and three columns.

7. The system of claim 1, wherein the classification of the first data set is implemented during execution of a machine learning engine application.

8. A method comprising: storing, by a memory, a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data; converting, by a processor, the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer; converting, by a processor, the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer; loading data elements of the third buffer and the fourth buffer from the memory into a cache through a plurality of computation units; performing convolution filter operations on the data elements of the third buffer and the fourth buffer by a plurality of computing units to generate results in the tiled format; as well as Classification of the first data set is achieved based on the results from the convolutional filter operation.

9. The method of claim 8, wherein the cache comprises a plurality of channels, and wherein the method further comprises loading convolution filter data from the third buffer on the plurality of channels in parallel within a single clock cycle through the plurality of computational units.

10. The method of claim 8, wherein the cache comprises a plurality of channels, and wherein the method further comprises loading the image data from the fourth buffer on the plurality of channels in parallel within a single clock cycle by the plurality of computing units.

11. The method according to claim 8, wherein: Converting the first buffer from the linear format to the third buffer in the tiled format comprises: reading values ​​from sequential locations of said first buffer in said memory; and Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

12. The method according to claim 8, wherein: Converting the second buffer from the linear format to the fourth buffer in the tiled format comprises: reading values ​​from sequential locations of said second buffer in said memory; and Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

13. The method of claim 8, wherein the convolution filter data comprises a plurality of convolution filters, wherein each of the plurality of convolution filters has three rows and three columns.

14. The method of claim 8, further comprising implementing the classification of the first data set during execution of a machine learning engine application.

15. An apparatus comprising: a memory storing a first buffer and a second buffer in a linear format, wherein the first buffer stores convolution filter data and the second buffer stores image data; as well as A processor configured to: converting the convolution filter data in the first buffer from the linear format to a tiled format and storing in a third buffer; as well as converting the image data in the second buffer from the linear format to the tiled format and storing in a fourth buffer; as well as Classification of the first data set is achieved based on results from convolution filter operations performed on data elements stored in the third buffer and the fourth buffer in a tiled format.

16. The apparatus of claim 15, further comprising a cache, wherein the cache comprises a plurality of channels, and wherein the apparatus is further configured to load the convolution filter data from the third buffer in parallel on the plurality of channels within a single clock cycle.

17. The apparatus of claim 15, further comprising a cache, wherein the cache comprises a plurality of channels, and wherein the apparatus is further configured to load the image data from the fourth buffer in parallel on the plurality of channels within a single clock cycle.

18. The device according to claim 15, wherein: Converting the first buffer from the linear format to the third buffer in the tiled format comprises: reading values ​​from sequential locations of said first buffer in said memory; and Stepping through the third buffer with a stride equal to the sum of the first number of input channels and the second number of convolution filters while writing the values ​​to the third buffer.

19. The device according to claim 15, wherein: Converting the second buffer from the linear format to the fourth buffer in the tiled format comprises: reading values ​​from sequential locations of said second buffer in said memory; and Stepping through the fourth buffer with a stride equal to a number of pixel channels, simultaneously writes the values ​​to the fourth buffer.

20. The apparatus of claim 15, further comprising implementing the classification of the first data set during execution of a machine learning engine application.

Images