Accelerated mathematical engine

Abstract

To provide a system and a method with a high calculation throughput capable of promptly and efficiently executing a matrix mathematical operation.SOLUTION: A system 200 applies an accelerated mathematical engine to image processing in such a way that convolution of an image is accelerated by using a two-dimensional matrix processor 240 which includes sub-circuits 242 that include an Arithmetic Logical Unit (ALU), an output register and a shadow register. This architecture supports a clocked, two-dimensional architecture in which image data and weights are multiplied in a synchronized manner to allow a large number of mathematical operations to be performed in parallel.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 62 / 536,399 (20150-2154P(P0822-1PUS)), entitled "Accelerated Mathematical Engine," filed July 24, 2017, and lists Peter Joseph Bannon, Kevin Altair Hurd, and Emil Talpes as inventors. The aforementioned patent documents are incorporated herein by reference in their entireties.

[0002] This application also claims priority to commonly owned U.S. Patent Application No. 15 / 710,433 (20150-2154(N0822-1NUS)), entitled "Accelerated Mathematical Engine," filed September 20, 2017, and listing Peter Joseph Bannon, Kevin Altair Hurd, and Emil Talpes as inventors. Each of the foregoing patent documents is incorporated herein by reference in its entirety.

[0003] The present disclosure relates to an accelerated math engine that operates on large amounts of data, and more particularly to an accelerated math engine that performs complex convolution operations based on matrix multiplication operations. [Background technology]

[0004] Those skilled in the art will recognize the ever-increasing demands for speed and performance from typical processors and systems used to perform time-sensitive, complex mathematical operations. Because these typical systems are used to process large amounts of data and perform complex mathematical operations, their computational resources and speed are limited by the capabilities of existing, common hardware designs that perform these calculations. For example, general-purpose computer devices and processors that perform matrix operations may not be able to perform these operations in a timely manner under certain circumstances. Many conventional multipliers that perform digital signal processing operations rely on a series of software and hardware matrix manipulation procedures (e.g., address generation, transposition, bitwise addition and shifting), which can become a bottleneck in time-sensitive systems. Often, these manipulation steps require the use of processor arithmetic functions to generate intermediate results, but the intermediate results must be saved and fetched from various locations to complete the operation, resulting in wasted computation time.

[0005] 1 shows an example of a conventional multiplier system. Multiplier system 100 is a scalar machine that includes a computation unit 102, registers 104, a cache 106, and a memory 108. During operation, computation unit 102 uses registers 104 and cache 106 to retrieve data stored in memory 108. Typically, computation unit 102 is a microprocessor, such as a CPU or GPU, that can perform various computational procedures, including matrix multiplication on input matrices to obtain a result matrix, for example, by converting multiplications to additions and outputting the results to some internal register.

[0006] For example, a dot product representing an output pixel of an image is typically generated by dot-multiplying individual matrix elements of two matrices to obtain partial results and then adding the final results to obtain the final dot product. Multiplication of individual matrix elements, or scalar multiplication, is typically performed on individual data elements by breaking the dot multiplication into a series of individual sub-operations. As a result, partial products must be stored and fetched from one or more of registers 104, cache 106, and memory 108 to complete a single arithmetic operation.

[0007] Applications requiring computations such as convolution often require software functions built into the compute unit 102 that are used to convert the convolution operation into an alternative matrix multiplication operation. This is accomplished by rearranging and reformatting the data into two matrices that can then be directly multiplied. However, there is no mechanism for efficiently sharing or reusing data in the scalar machine 100, as the data required to perform each scalar operation must be repeatedly restored and re-fetched from registers. The complexity and management overhead of these operations grows significantly as the amount of image data being convolved increases.

[0008] The inability to reuse much of the data in scalar machine 100, coupled with the added and inefficient steps of storing and fetching intermediate results from registers 104, cache 106, and memory 108 to complete arithmetic operations, are just some of the drawbacks of existing systems such as multiplier system 100. Summary of the Invention [Problem to be solved by the invention]

[0009] Therefore, what is needed is a high computational throughput system and method that can perform matrix mathematical operations quickly and efficiently. [Brief explanation of the drawings]

[0010] Reference may be made to embodiments of the present invention, examples of which are illustrated in the accompanying drawings. These figures are for purposes of illustration and not limitation. While the invention will generally be described in the context of these embodiments, it will be understood that it is not intended to limit the scope of the invention to these particular embodiments. Items in the figures may not be to scale.

[0011] [Figure 1] 1 shows an example of a conventional multiplier system.

[0012] [Figure 2] 1 illustrates an exemplary matrix processor architecture for performing arithmetic operations in accordance with various embodiments of the present disclosure.

[0013] [Figure 3] 3 illustrates details of an exemplary configuration of the matrix processor architecture shown in FIG. 2.

[0014] [Figure 4] 4 illustrates an exemplary multiply-add circuit implementation of the logic circuit shown in FIG. 3.

[0015] [Figure 5] 1 illustrates an exemplary convolution operation according to various embodiments of the present disclosure.

[0016] [Figure 6] 1 illustrates details of an exemplary convolution operation according to various embodiments of the present disclosure. [Figure 7] 1 illustrates details of an exemplary convolution operation according to various embodiments of the present disclosure. [Figure 8] 1 illustrates details of an exemplary convolution operation according to various embodiments of the present disclosure.

[0017] [Figure 9] 1 illustrates an exemplary deconvolution operation according to various embodiments of the present disclosure.

[0018] [Figure 10]FIG. 1 illustrates a process for performing arithmetic operations to make a convolutional neural network faster, according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] In the following description, for purposes of explanation, specific details are set forth to provide an understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without these details. Furthermore, those skilled in the art will recognize that the embodiments of the invention described below can be implemented in various ways, such as a process, an apparatus, a system, a device, or a method on a tangible computer-readable medium.

[0020] The configurations or modules shown in the figures are illustrative of exemplary embodiments of the present invention and are intended to avoid obscuring the present invention. Also, throughout this description, configurations are described as separate functional units that may include subunits, but those skilled in the art will recognize that various configurations, or portions thereof, may be divided into separate configurations or integrated together, including integrated within a single system or configuration. Note that the functions or operations described herein may be implemented as configurations. Configurations may be implemented in software, hardware, or a combination thereof. Many configurations are formed through the interconnection of many subconfigurations. Subconfigurations that operate logically differently from those shown herein may be selected (e.g., an active high signal may be replaced with an active low signal, an AND gate may be replaced with an inverted-input NOR gate, etc.) if these logically distinct subconfigurations can be aggregated and combined with other subconfigurations to provide similar or identical functionality at the aggregate configuration level as described herein.

[0021] Furthermore, connections between structures or systems in the figures are not limited to direct connections. Rather, data between these structures may be modified, reformatted, or altered by intermediate structures. Also, additional or fewer connections may be used. Also, it should be understood that the terms "coupled," "connected," or "communicatively coupled" include direct connections, indirect connections through one or more intermediary devices, and wireless connections.

[0022] References herein to "one embodiment," "a preferred embodiment," "an embodiment," or "some embodiments" mean that a particular feature, structure, characteristic, or function described in connection with an embodiment is included in at least one embodiment of the invention, and may be included in more than one embodiment. Also, appearances of such phrases in various places in the specification do not necessarily all refer to the same embodiment or embodiments.

[0023] The use of particular terms in various places herein is for purposes of explanation and should not be construed as limiting. A service, function, or resource is not limited to a single service, function, or resource; use of these terms refers to a grouping of related services, functions, or resources, which may be distributed or aggregated.

[0024] The terms "comprises," "including," "comprises," and "having" are understood to be open terms, and any lists below are examples and are not intended to be limited to the listed items, but may include subsets or supersets of items, as well as additional items. Any headings used herein are for organizational purposes only and are not to be used to limit the scope of the description or any claims. Each document cited in this patent document is incorporated herein by reference in its entirety.

[0025] Additionally, one skilled in the art should recognize that (1) certain steps may be performed optionally, (2) steps may not be limited to the particular order described herein, (3) certain steps may be performed in a different order, and (4) certain steps may be performed simultaneously.

[0026] Although the embodiments herein are described primarily in the context of convolution, those skilled in the art will appreciate that deconvolution and other matrix operations can also be configured as matrix-matrix multiplication operations, and thus the principles of the present invention are similarly applicable to deconvolution. Additionally, other types of mathematical operations may be performed in accordance with various embodiments of the present disclosure.

[0027] FIG. 2 illustrates an exemplary matrix processor architecture for performing arithmetic operations in accordance with various embodiments of the present disclosure. System 200 includes logic circuits 232, 234, cache / buffer 224, data formatter 210, weight formatter 212, data input matrix 206, weight input matrix 208, matrix processor 240, output array 226, post-processing unit 228, and control logic 250. Matrix processor 240 includes multiple subcircuits 242, including arithmetic logic units (ALUs), registers, and, in some embodiments, encoders (e.g., Booth encoders). Logic circuit 232 may represent N input operators and data registers. Logic circuit 234 may input M weight operands to matrix processor 240. Logic circuit 232 may input image data operands to matrix processor 240. Weight input matrix 208 and data input matrix 206 may be stored in various types of memory, including SRAM devices. Those skilled in the art will recognize that various types of operands may be input to matrix processor 240 .

[0028] In operation according to certain embodiments, system 200 accelerates convolution operations by reducing redundant operations within the system and implementing hardware-specific logic to perform certain mathematical operations across large sets of data and weights. This acceleration is a direct result of the manner in which image data and weights are obtained and input to matrix processor 240 (and the corresponding hardware configuration), as well as extensive timing of mathematical operations within matrix processor 240.

[0029] In an embodiment, the example formatters 210, 212 in FIG. 2 are implemented as inline formatters. In certain embodiments, the formatters 210, 212 are separate components, while in other embodiments, the formatters 210, 212 are integrated together and / or with one or more other components. Each is implemented in hardware, converting matrices into vectors on operands and operating within the matrix processor 240. In other embodiments, the formatters 210, 212 are implemented in software, which typically results in a slower speed. The data formatter 210 converts the two- or three-dimensional (e.g., a 3x3x3 cube) data comprising the data input matrix 206 into a single vector or string represented by rows or columns, thereby linearizing or vectorizing the data input matrix 206. Specifically, the formatter 210 receives the data input matrix 206 and prepares the input data to be processed by the matrix processor 240. In an embodiment, this is achieved by mapping the parameters of the data input matrix 206 into an appropriate format according to the hardware requirements of the matrix processor 240 so that the matrix processor 240 can efficiently perform the matrix multiplication as part of the convolution calculation when generating the output pixels.

[0030] As an example, assume that matrix processor 240 includes 96 rows and 96 columns. Data mapped to a 96x96 format allows matrix processor 240 to utilize its full computational capabilities, thus providing favorable efficiency. Formatter 210 must then generate a 96-column-wide output. Similarly, formatter 212 must generate a 96-row-wide output based on weight input matrix 208.

[0031] In an embodiment, the formatter 210 uses several multiplexers or switches to fetch some or all of the data input matrix 206 and select different elements therefrom to generate data ordered according to columns for the matrix processor 240. In an embodiment, the selection ensures that the appropriate data from the data input matrix 206 is passed to each column in a prescribed clock cycle. In an embodiment, if the weights are static, they may be pre-formatted offline, stored in memory, fetched only once, and provided directly to the matrix processor 240 in a modified vectorized format without using the formatter 212. In other embodiments, the weights may be dynamically adjusted and provided to the matrix processor 240 according to various formats and fetch operations. In an embodiment, the matrix processor 240 allows for input of columns and rows of various sizes. That is, the matrix processor 240 is designed to compute N×M calculations of any size.

[0032] In another embodiment, if the number of columns of matrix processor 240 is limited (e.g., to N columns) such that the number of columns (e.g., X) of data input matrix 206 is greater than the number of columns of matrix processor 240 (i.e., X>N), control logic 250 may divide data input matrix 206 into multiple sub-matrices, with each sub-matrix being calculated by matrix processor 240. In such a case, each matrix processor 240 may execute in a different thread. For example, if data input matrix 206 is composed of 192x96 data points and the matrix processor has 96 columns and 96 rows (i.e., 96x96 calculations occur in one clock cycle), control logic 250 may divide data input matrix 206 into two sub-matrices (e.g., the left half of data input matrix 206 and the right half of data input matrix 206), each sub-matrix being composed of 96x96 data points. Each independently threaded matrix processor 240 computes the output channel of the sub-matrix it is sent and places the results in a final output array 260 that should be large enough to hold the values ​​from all channels (i.e., 192 values). More generally, the data input matrix 206 may be split into any number of sub-matrices and sent to different matrix processors 240, each running in a separate thread. Like the output array 226, the data input matrix 206, data formatter 210, cache / buffer 224, logic circuitry 232, and post-processing unit 228 must be able to accommodate larger data as well.

[0033] In an alternative embodiment, the control logic 250 can divide the computation along dot products to compute the CNN among multiple matrix processors 240. Segments of the dot products are computed in different matrix processors 240, and the input products are then added together to compute an output vector, which is stored in an output array 260.

[0034] Unlike typical software implementations of formatting functions performed by a CPU or GPU to convert convolution operations into matrix multiplication by rearranging data into an alternative format suitable for fast matrix multiplication, various hardware implementations of the present disclosure allow data to be formatted and executed on the fly. For example, 96 pieces of data are valid per cycle, allowing a very large number of matrix elements to be processed in parallel, thereby efficiently mapping data to matrix operations. In embodiments, for 2N fetched input data, 2N 2 of computational data can be obtained in a single clock cycle. This architecture not only provides a parallel, efficient, and synchronous process for performing numerous mathematical operations across multiple data inputs, but also significantly improves processing speed by effectively reducing the number of read or fetch operations used in typical processor architectures.

[0035] In embodiments, to increase the efficiency of matrix processor 240 with any number of columns and rows, formatters 212, 214 can reformat different shapes of input matrix data into columns and rows suitable for matrix processor 240. In embodiments, formatting is performed dynamically to accommodate processing of matrices with different input sizes. In embodiments, the reformatted matrix containing the input channels is provided to cache / buffer 224.

[0036] The cache / buffer 224 may fetch data from the data input matrix 206 1 / k times, where k is the convolution kernel width, as various pieces of data are reused. For example, when a row is fetched in a particular cycle, a particular column has access to all of the data for that row. In some embodiments, the cache / buffer 224 may be a partial buffer that stores partial copies of data that can be reused by the convolution without having to re-access and read data from the SRAM.

[0037] Once matrix processor 240 completes its calculations, a set of results may be shifted, for example, from the accumulators in the bottom row of matrix processor 240 to output flip-flops (not shown) that effectively form a shift register that receives, for example, the dot products. In an embodiment, pulling or shifting the results from the rows corresponding to the output channels to output array 226, for example, one per clock cycle, may be accomplished by a state machine (not shown). The state machine may perform additional operations on the output channels, for example, before sending the data to SRAM and / or post-processing unit 228. The internal operation of matrix processor 240 is described in more detail below.

[0038] In an embodiment, matrix processor 240 includes shadow registers that enable parallel processing by storing copies of results that are passed through matrix processor 240 to output array 226. In an embodiment, moving the results of an operation from the output registers to the shadow registers involves loading the next set of values ​​into the ALU.

[0039] Once the accumulation is complete, the convolution begins and may be redone before all data from the previous convolution is output to the output array 226. As a result, with each clock cycle, the data in matrix processor 240 moves down one row, so that with each cycle the last row is output to the output array 226. In effect, this mode of operation ensures that a new calculation occurs with each successive cycle without interruption, independent of additional processing operations such as saving data to SRAM.

[0040] Post-processing unit 228 may include or interact with a number of devices (not shown), such as a hardware accelerated pooling unit, DRAM, which may be part of a direct memory access ("DMA") that retrieves data from memory and stores data (such as weights and results) in SRAM, etc. The devices may be partially or wholly controlled by control logic 250, which manages formatters 210, 212 and other configurations within system 200.

[0041] Not shown in FIG. 2 are auxiliary devices that perform management functions, such as a sequencer that keeps track of where system 200 is in the convolution, generating addresses for reading data, writing results, and calculating where to get and how to perform data used in subsequent steps of the convolution.

[0042] In particular embodiments, the weight input matrix 208 is physically split to drive weights from two different sides of the matrix processor 240, such that the two-dimensional array is divided into two regions (e.g., left and right sides), each receiving a portion of the data in the weight input matrix 208. Such an implementation reduces data latency by leveraging the fact that the weights are known. In embodiments, to reduce peak power consumption, operational timing may be selected such that multiplications of weights and data are distributed over a particular number of cycles. This efficient operational timing reduces energy consumption steps, such as reducing the number of read operations performed by the matrix processor and improving the efficiency of data movement within the matrix (e.g., between subcircuits).

[0043] In an embodiment, a state machine (not shown) configured to identify redundant data may be used. The identified redundant data may be reused across columns, eliminating the need to re-fetch data. The state machine may be configured to determine how and where to shift the data based on inputs related to, for example, image size, filter size, stride, number of channels, and similar parameters.

[0044] In an embodiment, a Booth encoder is shared across several elements of the multiplication architecture of matrix processor 240. The Booth encoder may be any Booth encoder known in the art and may be used to multiply two numbers and encode one of the two numbers, for example, from an 8-bit value to a 12-bit or other value that facilitates the multiplication operation in the multiplier logic and is therefore faster. In an embodiment, the Booth encoder may be applied in parallel across an entire row to share the same coded alternate weight value across all columns. By loading operands into all columns, multiplications can be performed across an entire row in one clock cycle. Thus, the cost of utilizing re-encoding to share the same data (e.g., weights) across N computational elements is paid only once for each column (or row). In contrast, in existing computational architectures, a Booth encoder is required for every multiplication operation for every single scalar.

[0045] FIG. 3 shows details of an exemplary configuration of the matrix processor architecture shown in FIG. 2. In an embodiment, the matrix processor 300 may accommodate a predetermined vector length on each axis. As shown in FIG. 3, the matrix processor 300 may comprise an array of 6×6 tiles 302 arranged in a matrix format. Each tile 302 may include a matrix 320 that in turn includes sub-circuits 350. As described in more detail below with reference to FIG. 4, each sub-circuit 350 may be a cell capable of performing arithmetic operations. In an embodiment, the sub-circuits 350 simultaneously perform multiplication, accumulation, and shift operations.

[0046] In an embodiment, arithmetic operations are parallelized by utilizing multiple rows and columns of matrix processor 300 to generate an NxN tile output. For example, a row size of 96 and a corresponding column size of 96 facilitates the output of 2*9216 mathematical calculations. In other embodiments, the number of rows and columns may be different, i.e., N rows and M columns, generating an NxM tile output. For example, a row size of 96 and a corresponding column size of 192 would generate 2*18,432 calculation results in one clock cycle.

[0047] Figure 4 shows an exemplary multiply-add circuit implementation of the sub-circuit shown in Figure 3. As shown in Figure 4, the multiply-add circuit 400 includes a multiplier 430, an adder 432, logic 434, 436, 438, an accumulator 424, a shadow register 428, and an output register 440. In an embodiment, the accumulator 424 may be implemented as an accumulation register.

[0048] In an embodiment, the accumulator 424 may include a set of ALUs that include registers and a shadow register 428 that may be configured to receive the output of the ALUs.

[0049] In operation, multiplier 430 receives and multiplies weights 402 and data 404 to generate products therefrom. Each product may be provided to adder 432, which, in response to receiving the product from multiplier 430, adds the product to the current value of accumulator 424.

[0050] In an embodiment, accumulator 424 generates an accumulated value that is stored, for example, in output register 440. The accumulated value is the result of a convolution and may correspond to the dot product of two formatted matrices, as described with reference to FIG.

[0051] In an embodiment, a copy of the result in output register 440 is provided to shadow register 428 which outputs result 450, and accumulator 424 can be accessed again to begin a new calculation. In an embodiment, multiply-add circuit 400 of FIG. 4 can perform multiplication, addition, and shift operations simultaneously, i.e., within a single cycle, thereby doubling the total number of operations occurring each cycle.

[0052] In an embodiment, the ClearAcc signal 408 clears the contents of the accumulator 424 so that the accumulation operation can be redone, for example, when the multiplier 430 performs a multiplication operation. In an embodiment, the ResultEnable signal 412 is activated in response to determining that the data 404 is valid. It will be appreciated that the accumulator 424 may accumulate and store data, accumulate and clear data, or simply clear data.

[0053] In an embodiment, the result is moved from the output register 440 to the shadow register 428 in a single clock cycle, i.e., without the need for intermediate execute and store operations.

[0054] 5 illustrates an exemplary convolution operation according to various embodiments of the present disclosure. The convolution 500 includes input channels IC of an input image 502, weights 532, a dot product 514, output channels OC, and an accumulator 540.

[0055] In an embodiment, the convolution operation 500 applies individual filters (i.e., weights) 532 to the input image 502, for example, to detect small features within the input image 502. By analyzing a series of different features in different orders, macro-features can be identified in the input image 502. In other embodiments, the input 502 is non-image data. For example, the input 502 may be non-image sensor data, such as ultrasound, radar, lidar, or other sensor data. The input 502 may also be general mathematical calculations or other types of data known to those skilled in the art.

[0056] The convolution 500 may use a different set of weights 532 for each input channel IC because each input channel IC may contain a different set of information, and each weight matrix 532 may be designed to help identify different features. In an embodiment, the convolution 500 multiplies a rectangular input matrix 504 by a rectangular weight matrix 532 to obtain partial dot products. The partial dot products may then be summed by an adder 546 to generate an accumulated dot product 514 (i.e., an integer) that represents an output pixel 514 in the output image.

[0057] In an embodiment, each pixel of output channel OC is generated by a multiplier 542 and an adder 544. In an embodiment, the value of the partial dot product corresponds to the application of the entire weight matrix 532 to a region 504 of input image 502. In other words, each weight 532 is dot-multiplied with region 504 by multiplier 542 to generate a partial dot product, which is then accumulated in accumulator 540 to generate an accumulated output representing the convolution.

[0058] One or more input channels IC may be used, e.g., one for each color (e.g., RGB). For example, each convolution may use weights 532 representing three different matrices, one for each color. Each output channel OC 512 may be generated using a different filter or weights 532 representing a different feature of the input data 502. The number of output channels may depend on the number of functions. The number of convolutions is equal to the number of output channels OC multiplied by the number of input channels IC, and each convolution may have N convolutions for each input channel IC. Those skilled in the art will recognize that the number and type of input channels may vary and may include color and / or clear inputs.

[0059] 5, input matrix 504 is a Kx×Ky (i.e., 3×3) matrix that can be combined with a 3×3 weight matrix 532 across three input channels, i.e., 3×3×IC, depth-matching to produce a single element, dot product 514, in the output plane. Each dot product 514 in output channel 512 is the result of a dot multiplication.

[0060] 6-8 show details of an exemplary convolution operation according to various embodiments of the present disclosure. Convolution 600 includes an input data matrix 602, a weight data matrix 604, an array 606, and an inner product 630. In an embodiment, array 606 is a matrix processor architecture such as that shown in FIGS. 2 and 3.

[0061] Input data matrix 602 of FIG. 6 includes columns 610 that, in embodiments, can be obtained by linearizing an input matrix, such as rectangular input matrix 504 shown in FIG. 5, to obtain a vectorized form of the input matrix. Similarly, weight data matrix 604 includes rows 620 that can be a vectorized form of a weight matrix, such as rectangular weight matrix 532 of FIG. 5. As an example, a 3×3 input matrix and three input channels can be reformatted into a vector containing 3×3×3 = 27 elements, which can generate the 27-element column 610 for use in input data matrix 602. Conversely, the same 3×3 weight matrix of three input channels can be used to generate the 27-element row 620 for use in weight data matrix 604. Those skilled in the art will recognize that the size of the input matrix and the number of input channels can vary across different applications.

[0062] In an embodiment, the input channels and input weights, depicted as rectangles in Figure 5, are reformatted, for example by the formatter discussed with reference to Figure 2, into a vector format (e.g., a vector having 96 elements) that is provided to a matrix multiplier / processor (shown as element 240 in Figure 2) that can perform 96x96 element dot product operations in parallel. In particular, the input data 504 and input weights 532, depicted in Figure 5 as rectangles for each input channel, are reformatted into vector form.

[0063] 6 as input data 602 and input weights 604 (e.g., each containing 96 elements) are provided to a matrix processor or matrix multiplier 240, which performs 96×96 element dot product operations in parallel. In an embodiment, the calculation of an output channel uses the same set of input data but a different set of weights (i.e., a filter) to generate the same output pixel, and many output channels can be generated at once by reading the input data. As noted above, it is understood that the number of input channels and the number of output channels can be selected arbitrarily.

[0064] Further, it is understood that input data matrix 602, weight data matrix 604, and array 606 may have a different number of columns and rows than those shown in Figure 6. In particular, the shapes of input data matrix 602 and weight data matrix 604 may be formatted to correspond to the columns and rows of any arbitration configuration of array 606. Additionally, in situations where weight data matrix 604 is known, rows 620 can be generated and stored in a vectorized format without the use of a formatter.

[0065] 6 is generated by dot-multiplying the vector corresponding to column 610 with the vector corresponding to row 620. In an embodiment, as shown in FIG. 7, the next dot product 632 may be obtained by dot-multiplying the vector corresponding to column 612 with the vector corresponding to row 620. As one skilled in the art will recognize, once all dot products for the first row of array 606 have been filled, the dot product for the second row of array 606 may be calculated by dot-multiplying the elements of the first column 610 of input data matrix 602 with the second row of weight data matrix 604, and so on.

[0066] It is important to note that Figures 6-8 are for illustrative purposes only, and the dot multiplications described above can be performed simultaneously to produce a one-shot matrix-matrix multiplication operation.

[0067] FIG. 9 illustrates an exemplary deconvolution operation according to various embodiments of the present disclosure. The deconvolution system 900 includes an input channel IC of an input image 902, weights 922, dot products 904 and 906, and an output channel OC. Those skilled in the art will recognize that the deconvolution operation 900 is actually the mathematical transpose (nearly inverse) of a convolution operation, such as the convolution shown in FIG. 5. Those skilled in the art will further recognize that the deconvolution operation 900 can be learned using a neural network by applying procedures similar to those used in conventional convolutional neural networks. For the sake of brevity, the description or functionality of configurations similar to those in FIG. 5 will not be repeated here.

[0068] In an embodiment, the deconvolution operation 900 of Figure 9 reconstructs the matrix 912 by decomposing the dot product 904906 using weights 922. Similar to a convolution operation, the deconvolution 900 may use a different set of weights 922 for each input channel IC. In an embodiment, the deconvolution 900 may be advantageously applied to images to perform image deconvolution, for example, to improve robustness to artifacts. Other applications may include image data analysis and restoration, etc.

[0069] FIG. 10 illustrates a process for performing arithmetic operations to accelerate a convolutional neural network according to various embodiments of the present disclosure.

[0070] The process 1000 for performing arithmetic operations begins at step 1002 when a first set of operands capable of representing rows of a data matrix is ​​received from a first logic circuit. This first set of operands can be vectorized such that the operands are aligned with the inputs to the matrix processor. In certain embodiments, the size of the vectorized operands is directly related to the number of inputs to the matrix processor along an axis.

[0071] In step 1004, a second set of operands, which may represent columns of a weight matrix, is received from the second logic circuit. This second set of operands is vectorized such that the operands are aligned within corresponding inputs to the matrix processor. In certain embodiments, the size of the vectorized operands is directly related to the number of inputs to the matrix processor along different axes.

[0072] In step 1006, the first set of operands is dot-multiplied with the second set of operands to obtain one or more dot products. In particular embodiments, this set operation across the sets of operands is performed in a single clock cycle.

[0073] In step 1008, the image may be convolved with the filter using a dot product to generate a convolution result.

[0074] The convolution results are further processed to enhance the image output in step 1010. This further processing may occur using non-linear functions, normalization operations, or pooling operations.

[0075] Those skilled in the art will recognize that the computing system or programming language is not critical to the practice of the present invention. Those skilled in the art will also recognize that many of the elements described above may be physically and / or functionally separated into sub-modules or combined together.

[0076] It should be noted that the elements of the following claims can be arranged in various ways, including including multiple dependencies, configurations, and combinations. For example, in embodiments, the subject matter of various claims can be combined with other claims.

[0077] Those skilled in the art will understand that the foregoing examples and embodiments are illustrative and do not limit the scope of the present invention. All substitutions, enhancements, equivalents, combinations, and improvements apparent to those skilled in the art upon reading the specification and studying the drawings are intended to be within the true spirit and scope of the present invention.

Claims

1. 1. A matrix processor for accelerating convolutions in a neural network, comprising: a first input circuit arranged in a first dimension of the two-dimensional array, the first input circuit coupled to receive N operands from the first logic circuit, the N operands formatted according to a first width associated with the first dimension; a second input circuit arranged in a second dimension of the two-dimensional array, the second input circuit coupled to receive M operands from a second logic circuit, the M operands formatted according to a second width associated with the second dimension; a plurality of sub-circuits coupled to receive the N operands and the M operands, at least a subset of the plurality of sub-circuits including an arithmetic logic unit, an accumulator, and a shadow register, the sub-circuits coupled in the two-dimensional array performing arithmetic operations on the N operands and the M operands; A matrix processor comprising:

2. The matrix processor of claim 1 , wherein the arithmetic operation is an inner product calculation for a convolution operation.

3. The matrix processor of claim 2 , wherein the arithmetic logic unit includes a multiply-add circuit that generates the inner product.

4. 2. A matrix processor as claimed in claim 1, wherein said N operands represent image data and said M operands represent weight values.

5. 2. The matrix processor of claim 1, wherein at least some of the sub-circuits include encoding elements configured to encode values ​​representing one or more of the M operands.

6. 6. A matrix processor according to claim 5, wherein said encoding element is a Booth encoder.

7. 2. The matrix processor of claim 1, wherein the N operands are formatted from a data input matrix.

8. The matrix processor of claim 1 , further comprising a state machine that uses at least one of a filter size and a stride to determine which of the N operands or the M operands are reusable.

9. The matrix processor of claim 1 , wherein the accelerated processing speed is achieved by reducing read operations from a cache and by accelerating data throughput through the plurality of sub-circuits.

10. 1. A system for accelerating convolutions in a neural network, comprising: a first logic circuit for generating N operands; a first input circuit arranged in a first dimension of a two-dimensional array and coupled to receive the N operands from the first logic circuit; a second logic circuit for generating the M operands; a second input circuit arranged in a second dimension of the two-dimensional array and coupled to receive the M operands from the second logic circuit; a matrix processor including a plurality of sub-circuits configured to perform dot multiplication of the N operands with the M operands to generate dot products; an output array coupled to the two-dimensional array configured to use the dot product to generate a result; A system comprising:

11. 11. The system of claim 10, wherein the N operands are formatted from a data input matrix into a first vector and the M operands are formatted from a weight input matrix into a second vector.

12. 12. The system of claim 11, wherein the first logic circuit includes a plurality of data registers for storing the N operands, the plurality of data registers having a first width corresponding to the first dimension of the two-dimensional array, and the second logic circuit includes a weight register for storing the M weight operands, the plurality of weight registers having a second width corresponding to the second dimension of the two-dimensional array.

13. The system of claim 12 , wherein the first width corresponds to a number of cycles to produce the result.

14. 13. The system of claim 12, wherein the data register and the weight register are accessed only once to fetch the first number of elements and the second number of elements, respectively.

15. 11. The system of claim 10, wherein the subcircuit comprises a shadow register configured to move data into a shift register in one or more clock cycles.

16. 11. The system of claim 10, further comprising a buffer coupled to at least one of the data input matrix and the weight input matrix, the buffer storing copies of recently used data to enable reuse without refetching in subsequent cycles.

17. The system of claim 10 , wherein the result is an output matrix corresponding to the application of a filter to a region of an image.

18. The system of claim 10 , further comprising a state machine that uses at least one of a filter size and a stride to identify reusable data.

19. 1. A method for accelerating a convolutional neural network using a matrix multiplication circuit, comprising: receiving a first set of operands representing rows in a data matrix from a first logic circuit; receiving a set of second operands representing columns of a weight matrix from a second logic circuit; dot-multiplying the first set of operands with the second set of operands to obtain one or more dot products, and convolving the image with a filter using the dot products to generate a convolution result; A method comprising:

20. 20. The method of claim 19, wherein convolving the image comprises processing the one or more dot products through a convolutional layer to generate a layer output.

21. 21. The method of claim 20, wherein generating the layer output comprises applying one of a nonlinear function, normalization, and pooling to the convolution result.

Images