Ultrapipeline Accelerator for Machine Learning Inference

Abstract

To provide a method of pipelining inference of a neural network.SOLUTION: A method of pipelining inference of a neural network comprising an i-th layer, (i+1)-th layer, and (i+2)-th layer is provided. The method comprises: processing a first set of i-th values of the i-th layer to generate (i+1)-th values for the (i+1)-th layer; determining the quantity of the (i+1)-th values sufficient for processing; and, upon the determination, processing the (i+1)-th values to generate an output value for the (i+2)-th layer while concurrently processing a second set of i-th values of the i-th layer, where i represents an integer greater than 0.SELECTED DRAWING: Figure 2a

Description

[Technical field]

[0001] Some embodiments of the present invention relate generally to machine learning. [Background technology]

[0002] The explosive growth of big data-based machine learning (ML) applications, along with the prospect of a slowdown in Moore's Law, is prompting research into application-specific hardware fabrics. With a focus on embedding computing inside the memory bitcell, Processing-in-Memory (PIM) has been proposed to accelerate machine learning inference applications. Machine learning applications and networks must be efficiently mapped onto the underlying hardware fabric to extract the best power performance.

[0003] The above information disclosed in the Background of the Invention section is intended to enhance the understanding of the present disclosure and may therefore contain information that does not constitute prior art known to one of ordinary skill in the art. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical problem to be solved by the present invention is to provide a method for pipelined inference of a neural network with no or few stalls.

[0005] The technical problem that the present invention seeks to solve is to provide a system for pipelined inference of neural networks with little or no stalls.

[0006] The technical problem to be solved by the present invention is to provide a processing-in-memory (PIM) system that performs pipelined inference of a neural network with no or few stalls.

[0007] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person skilled in the art from the following description. [Means for solving the problem]

[0008] Embodiments of the present invention relate to systems and methods for mapping machine learning (ML) applications to accelerator-based processing-in-memory (PIM) in a pipelined manner. According to some embodiments, pipelining may include inter-tier pipelining, intra-tier pipelining, and / or a combination of the two. Additionally, the pipelining schemes of various embodiments allow for the generation of multiple outputs in one clock cycle for multiple images simultaneously.

[0009] Some embodiments of the present invention relate to a configurable hardware accelerator based on mixed-signal PIM, where a machine learning problem is mapped to PIM sub-arrays in a pipelined manner.

[0010] Some embodiments of the present invention are directed to a configurable hardware accelerator based on a mixed signal PIM that receives as input multiple images for recognition. The input activations for the multiple images are received in a pipelined manner to the accelerator.

[0011] A method of pipelining inference for a neural network including multiple layers including an i-th layer, an i+1-th layer, and an i+2-th layer according to some embodiments of the present invention includes processing a first set of i-th values ​​of the i-th layer to generate i+1-th values ​​for the i+1-th layer, determining an amount of the i+1-th values ​​that is determined to be sufficient for processing, and in response to said determination, processing the i+1 values ​​while processing a second set of i-th values ​​of the i-th layer to generate result values ​​for the i+2-th layer, where i is an integer greater than 0.

[0012] In some embodiments of the present invention, the first set of i-th values ​​comprises pixel values ​​of an image.

[0013] In some embodiments of the present invention, the method wherein the first set of i-th values ​​comprises values ​​of a feature map generated by a previous layer of the neural network.

[0014] In some embodiments of the invention, processing the first set of i-th values ​​for the i-th layer includes applying an i-th filter associated with the i-th layer to the first set of i-th values ​​for the i-th layer to generate an i+1-th value for the i+1-th layer.

[0015] In some embodiments of the present invention, the i th filter is a sliding convolutional filter of p×q matrix form, where p and q are integers equal to or greater than 0.

[0016] In some embodiments of the invention, applying the i-th filter includes performing a matrix multiplication between the i-th filter and one of the first set of i-th values ​​that overlaps the i-th filter.

[0017] In some embodiments of the present invention, determining the amount of the i+1th value that is deemed sufficient to process includes determining that the i+1th filter associated with the i+1th layer has data for each unit operated on by the i+1th filter.

[0018] In some embodiments of the present invention, determining the amount of i+1th values ​​that is deemed sufficient to process is based on a clock cycle determined by at least one of the size of the ith filter associated with the i+1th layer, the size of the i+1th filter associated with the i+1th layer, the stride of the i+1th filter, and the size of the image and feature map corresponding to the first and second sets of ith values ​​of the ith layer.

[0019] In some embodiments of the invention, processing the first set of i values ​​for the i layer includes applying an i_1 filter associated with the i layer to a first portion of the first set of i values ​​to generate a first region of i+1 values ​​for the i+1 layer, and simultaneously applying the i_1 filter, applying an i_2 filter associated with the i layer to a second portion of the first set of i values ​​to generate a second region of i+1 values ​​for the i+1 layer.

[0020] In some embodiments of the invention, the first and second portions of the first set of i-th values ​​have values ​​that overlap each other.

[0021] In some embodiments of the invention, the first and second portions of the first set of i values ​​have non-overlapping values.

[0022] In some embodiments of the present invention, the i_1th filter and the i_2th filter are sliding convolution filters with the same stride and the same dimension.

[0023] In some embodiments of the present invention, the i_1th filter and the i_2th filter have the same weight value.

[0024] A system for pipelining inference of a neural network including a plurality of layers including an i-th layer, an i+1-th layer, and an i+2-th layer according to some embodiments of the present invention includes a processor and a processor memory adjacent to the processor, the processor memory storing instructions that, when executed by the processor, cause the processor to operate by: processing a first set of i-th values ​​of the i-th layer to generate i+1-th values ​​for the i+1-th layer; determining an amount of the i+1-th values ​​that is determined to be sufficient for processing; and, responsive to the determination, processing the i+1 values ​​while processing a second set of i-th values ​​of the i-th layer to generate a result value for the i+2-th layer, where i is an integer greater than 0.

[0025] In some embodiments of the invention, processing the first set of i-th values ​​for the i-th layer includes applying an i-th filter associated with the i-th layer to the first set of i-th values ​​for the i-th layer to generate an i+1-th value for the i+1-th layer.

[0026] In some embodiments of the present invention, determining the amount of i+1th values ​​that is deemed sufficient to process is based on a clock cycle determined by at least one of the size of the ith filter associated with the i+1th layer, the size of the i+1th filter associated with the i+1th layer, the stride of the i+1th filter, and the size of the image and feature map corresponding to the first and second sets of ith values ​​of the ith layer.

[0027] In some embodiments of the invention, processing the first set of i values ​​for the i layer includes applying an i_1 filter associated with the i layer to a first portion of the first set of i values ​​to generate a first region of i+1 values ​​for the i+1 layer, and simultaneously applying the i_1 filter, applying an i_2 filter associated with the i layer to a second portion of the first set of i values ​​to generate a second region of i+1 values ​​for the i+1 layer.

[0028] A configurable processing-in-memory (PIM) system for performing a neural network according to some embodiments of the present invention includes at least one first PIM subarray performing a filtering operation of an i-th filter of an i-th layer of the neural network, at least one second PIM subarray performing a filtering operation of an i+1-th filter of the i+1-th layer of the neural network, and a controller controlling the first and second PIM subarrays, wherein the controller provides a first set of i-th values ​​of the i-th layer to the at least one first PIM subarray to generate i+1-th values ​​for the i+1-th layer, determines an amount of the i+1-th values ​​determined to be sufficient for processing by the at least one second PIM subarray, and, responsive to said determination, provides the i+1-th values ​​to the at least one second PIM subarray while simultaneously providing a second set of i-th values ​​of the i-th layer to be processed to the at least one first PIM subarray for said processing, where i is greater than 0, to generate result values ​​for a subsequent layer of the neural network.

[0029] In some embodiments of the present invention, the PIM subarrays of the first and second PIM subarrays include a number of cells that store a number of weight values ​​corresponding to respective ones of the i th and i+1 th filters.

[0030] In some embodiments of the present invention, the neural network includes a convolutional neural network and the i-th filter is a sliding convolution filter. [Brief description of the drawings]

[0031] The features, other aspects and advantages of the embodiments of the present invention are explained in the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram illustrating a modifiable PIM system according to some embodiments of the present invention. [Figure 2a] FIG. 2 illustrates a configurable PIM system tile according to some embodiments of the present invention. [Figure 2b]FIG. 2 illustrates a PIM sub-array of the tile according to some embodiments of the present invention. [Figure 3a] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Figure 3b] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Figure 3c] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Figure 4a] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Figure 4b] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Figure 4c] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present invention. [Diagram 5] FIG. 2 illustrates batch pipelining of inference in a neural network according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The detailed description set forth below is intended as a description of embodiments of a system and method for accelerating pipeline machine learning, and is not intended to represent the only manner in which the present application may be constructed or utilized. The description discloses features of the present invention in connection with the illustrated embodiments. However, it is apparent that the same or equivalent functions and structures may be achieved by different embodiments that may be encompassed within the scope of the present application. Similar reference numerals in the drawings of the present application are intended to represent similar components and features.

[0033] The present application relates to mapping machine learning applications in a pipelined manner to a PIM-based accelerator. Pipelining may include inter-layer pipelining, intra-layer pipelining, or a combination of the two types. The PIM-based system is configurable, and the pipelining scheme is application specific and can be mapped to the PIM-based system on a case-by-case basis to increase (e.g., maximize) the power performance for each application. According to some embodiments, the pipelining scheme of the configurable PIM-based system can provide a significant (e.g., order of magnitude) power performance improvement over other digital or PIM-based inference accelerators of the related art. Additionally, the pipelining scheme can provide no or low latency operation of the network topology for the hardware.

[0034] FIG. 1 is a schematic diagram illustrating a configurable PIM system according to some embodiments of the present invention.

[0035] 1, once training is performed, the configurable PIM system 1 can infer input data to generate output data and make predictions based on the input data. According to some embodiments, the configurable PIM system 1 includes a PIM array 10 including multiple tiles 100 for performing inference operations, a controller 20 for controlling the operation of the PIM array 10, and a memory (30, e.g., on-logic-die memory) for storing result values ​​or intermediate results of each tile 100 of the configurable PIM system 1. In some embodiments, the memory 30 can be an embedded magnetoresistive random access memory (eMRAM), a static random access memory (SRAM), and / or the like.

[0036] Figure 2a is a diagram illustrating a tile 100 of a configurable PIM system 1 according to some embodiments of the present invention. Figure 2b is a diagram illustrating a PIM sub-array 110 of said tile 100 according to some embodiments of the present invention.

[0037] 2a, a tile 100 of a PIM array 10 according to some embodiments includes a plurality of PIM subarrays 110 in the form of a matrix, an input register 120 for receiving and storing an input signal Vin (e.g., also referred to as an input voltage signal) that may correspond to input data and providing the input signal stored in an appropriate one of the PIM subarrays 110, an analog-to-digital converter (ADC, 130) for converting an analog output of the PIM subarray 110 to a digital signal (e.g., a binary signal), a shift and add circuit 140 for storing and adding the output of the ADC 130 in one clock cycle, and an output register 150 for storing the output of the shift and add circuit 140 before transmitting the result to memory 30 for storage. In some embodiments, a controller 20 can control the operation of the input and output registers (120 and 150) and can determine which PIM subarray 110 to transmit the data stored in the input register 120 to.

[0038] Referring to FIG. 2b, each PIM sub-array 110 according to some embodiments includes a plurality of bit cells 112, which may be analog or digital (single or multi-bit). In some embodiments, the bit cells 112 may be two or three terminal non-volatile synapse weight value bit cells. The bit cells 112 may be resistive random access memory (RRAM) that may act as analog or digital memory. However, embodiments of the present application are not limited thereto, and the bit cells 112 in some embodiments may be conductive bridging random access memory (CBRAM), phase-change memory (PCM), ferroelectric field effect transistor (FerroFET), or spin transfer torque (STT) memory. The bit cells 112 may also include multiple units of memory cells and may have three or more terminals. To select individual bit cells 112, one or more diodes or field effect transistors (FETs) may be attached in series with the bit cells 112. The PIM sub-array 110 may further include peripheral circuitry such as a digital-to-analog converter (DAC, 114) that can convert a digital input into an analog voltage signal for application to one or more bit cells 112 and a sample and hold circuit 116 that stores the output of the bit cells 112 before transmitting to subsequent blocks.

[0039] In some embodiments, the PIM subarray 110 can operate as a filter (eg, a convolution filter) with each bit cell 112 storing a learnable weight (w1, w2, w3, etc.) of the filter.

[0040] 2a-b, a machine learning system may be described by a network such as a convolutional neural network (CNN), a fully connected (FC) neural network, and / or a recurrent neural network (RNN) having multiple interconnected layers. Such interconnected layers may be described by a weight matrix that is mapped to and stored in the bit cells 112.

[0041] In some embodiments, a set of PIM sub-arrays 110 is assigned to each layer of the neural network based on the network topology.

[0042] The number of rows of the PIM subarray 110 in the tile 100 may be determined by the size of the filters of the interconnected layers (e.g., layers of a CNN). The number of columns of the PIM subarray 110 in the tile 100 may be determined by the number of filters of that layer, the number of bits mapped per bit cell 112, and the precision of the filter weights. In some embodiments, each row of the PIM subarray 110 represents a filter of the CNN. Input activity of a previous layer may be provided concurrently (e.g., simultaneously) in one clock cycle as row inputs to the rows of the PIM subarray 110. In some embodiments, the rows and columns of the tile 100 may be shifted relative to one another as long as the logic in the controller 20 is changed.

[0043] In some embodiments, the PIM subarray 110 outputs a conductivity-weighted output current I out (which may represent synaptic neurons), and the output currents of the PIM subarrays 110 along a row are summed together (e.g., by a common electrical connection coupled to the output of the PIM subarrays 110). Thus, a row of the PIM subarrays 110 can generate I OUT1 Or I OUTN The summed output currents, which may be analog signals, are converted to a digital signal (e.g., binary data) by the ADC 130. Each summed output may represent a calculated partial sum, which is shifted by the shift and add circuit 140 and added to the partial sum of the next input activity. The output register 150 stores the calculated partial sums. Once the controller 20 determines that a final output has been generated, it may store the final output (back) in the memory 30 and then use it as an input to the next layer. The method and timing used to process this output data in the next layer may determine the pipelining scheme being implemented.

[0044] In some embodiments, the mapping of network layers to the PIM subarrays 110 is done in a pipelined manner, which may result in stall-free (or stall-reduced) operation. In some embodiments, rather than waiting for a neural network layer to complete processing before moving on to the next layer, the controller 20 begins processing the next layer of the neural network once sufficient output has been generated from the current layer. This is referred to as inter-layer pipelining (also known as inter-layer parallelism).

[0045] 3a-3c are diagrams illustrating inter-layer pipelining of inference in neural networks according to some embodiments of the present invention.

[0046] 3a-3c, a set of values ​​for the i-th layer can generate a two-dimensional matrix 300, where i is an integer greater than 0. In some embodiments, the two-dimensional matrix 300 represents an image, where each element 302 of the matrix 300 represents a pixel value (e.g., color intensity). The two-dimensional matrix 300 represents a feature map, where each element 302 represents an output value of a previous layer (i.e., the i-1th layer). For ease of explanation, the two-dimensional matrix 300 in FIGs. 3a-3c is a 6×6 matrix, and as will be appreciated by those skilled in the art, embodiments of the present application are not limited thereto, and the matrix 300 has any suitable size, represented as n×m, where n and m are integers greater than 1.

[0047] The ith filter (also called kernel) is a sliding convolution filter that operates on a set of values ​​in the ith layer to generate values ​​for the next layer (i.e., the i+1th layer). The filter 304 may be represented as a two-dimensional matrix of size p×q, where p and q are integers greater than 0 and less than, equal to, or smaller than n and m, respectively. For ease of explanation, the ith filter 304 is represented as a 2×2 filter. Each component of the convolution filter may be a learnable weight.

[0048] The i-th filter 304 shifts, slides, and moves across the set of values ​​of the i-th layer by a stride length until the entire set of values ​​of the i-th layer has passed. At each shift, the i-th filter 304 performs a matrix multiplication operation between the i-th filter 304 and the matrix portion of the i-th layer value on which the filter 304 is currently operating. Although the embodiment of Figures 3a-3c illustrates a stride length of 1, the stride may be 2, 3, or any suitable value. The output of the convolution operation that produces the set of values ​​of the next layer (i.e., layer i+1) may be referred to as the convolved characteristic output. Such output may fill the matrix 310 associated with the i+1-th layer.

[0049] Layers of neural networks in the related art need to be completely processed before values ​​for subsequent layers can be processed, which can be slow.

[0050] However, according to some embodiments, the controller 20 monitors the output values ​​generated by the i-th filter 304, and when it is determined that there are a sufficient number of values ​​available for processing in the next layer (i.e., the i+1-th layer), the controller 20 processes the output values ​​available in the i+1-th layer through the i+1-th filter 314. Thus, the controller 20 may apply the i+1-th filter 314 to the available values ​​of the i+1-th layer while simultaneously applying the i-th filter 304 to the values ​​of the i-th layer. Such inter-layer pipelining may be performed for any number or all of the neural network. That is, convolution operations may be performed simultaneously (e.g., may be performed simultaneously in two or more layers of the neural network). This may significantly improve the inference speed of the neural network.

[0051] According to some embodiments, at least one first PIM sub-array 110 performs the filtering operation of the i filter 304, and at least one second PIM sub-array 110 performs the filtering operation of the i+1 filter 314. The controller 20 may provide a first set of i values ​​of the i layer to the at least one PIM sub-array 110 to generate i+1 values ​​for the i+1 layer, and when it is determined that the amount of i+1 values ​​is sufficient to process the at least one second PIM sub-array 110, it may provide a second set of i values ​​for the processing to the first PIM sub-array 110 while providing the i+1 value to the second PIM sub-array 110 to generate output values ​​for a subsequent layer of the neural network. In some embodiments, it is determined that the amount of values ​​available for the filtering operation is sufficient when there is data for each device operated by the corresponding filter in the corresponding layer.

[0052] In some embodiments, the time (e.g., clock cycles) that the next layer (i.e., layer i+1) can begin processing may be dependent on the size of the i filter associated with the i layer, the size of the i+1 filter associated with the i+1 layer, the stride of the i+1 filter, and the size of the image or feature map corresponding to the i layer values. In the example of Figures 3a-3c, each of the i and i+1 filters 304, 314 has a size of 2x2, the i filter 304 has a stride of 1, and the i+1 filter 314 has a stride of 2, and the controller 20 begins processing the i layer and 7 cycles later begins processing the i+1 layer. As shown in Figure 3c, there are not enough values ​​available in the feature map 310 for the i+1 filter 314 to perform additional operations. According to some embodiments, each convolution operation may be performed in one clock cycle.

[0053] The speedup from inter-layer pipelining according to some embodiments of the present invention can be further improved by additionally using intra-layer parallelism.

[0054] 4a-4c are diagrams illustrating a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present invention.

[0055] With reference to FIGS. 4a-4c, in some embodiments, one or more filters at each layer of a neural network operate concurrently (e.g., simultaneously) to generate one or more output values ​​at a time.

[0056] In some embodiments, at a given time, the controller 20 applies an i_1th filter (e.g., an i_1th sliding filter, 304) associated with the ith layer to a first portion / block of the ith layer values ​​to generate a first output value for the i+1th layer (e.g., a first region of the i+1th values), and concurrently (e.g., simultaneously) applies an i_2th filter (e.g., an i_2th sliding filter, 305) associated with the ith layer to a second portion / block of the ith layer values ​​to generate a second output value (e.g., a second region of the i+1th values).

[0057] According to some embodiments, the i_1th and i_2th filters are identical (e.g., include the same weight map / weights) but offset in position by the length of the stride of the first filter 304. Thus, the i_2th filter 305 performs the same operation that the i_1th filter 304 did in the next clock cycle, but in the same clock cycle as the i_1th filter 304. Thus, the PIM array 10 in one clock cycle can generate more than one value for the next layer, where each layer can be subdivided and mapped to another tile with one or more copies of the same weight matrix. For example, the i_1th filter 304 can be implemented in one tile 100 and the i_2th filter 305 can be implemented in a different tile 100, whereby the weight matrix for the two tiles 100 is the same.

[0058] In some embodiments, the number of parallel operations performed by the filters in each layer may be equal to the number of filters that generate the combined / composite filter (e.g., 303) in that layer. Here, the stride of the composite filter may be equal to the number of filters that generate the composite filter (and perform parallel operations) multiplied by the strides of the included filters. For example, in the example of Figures 4a-4c, the stride of the composite filter 303, which includes two i-th filters (304 and 305) each having a stride of 1, is equal to 2. As will be appreciated by those skilled in the art, embodiments of the present invention are not limited to two parallel operations per layer, but may be extended to include any suitable number of parallel operations.

[0059] As shown in Figures 4a-4c, the controller 20 with inter-layer and intra-layer pipelining (combination of "ultra-pipelining") can start processing the i+1th filter after 4 clock cycles and continue processing the i+1th layer at the next clock cycle (t=5) without delay. This represents an improvement when compared to the inter-layer pipelining scheme of the embodiment of Figures 3a-3c, where the controller 20 can start processing the i+1th filter after 7 clock cycles and cannot perform the next filtering operation of the i+1th filter at the next clock cycle (t=8) due to an insufficient amount of values ​​available in the i+1th layer.

[0060] Although intra-layer pipelining / parallelism has been described for the i-th layer, according to some embodiments, the PIM array 10 can leverage intra-layer pipelining / parallelism at one or more layers (e.g., all layers) of the neural network.

[0061] In addition to using inter-layer and intra-layer pipelining to improve the processing speed of a single image or feature map, embodiments of the present invention utilize batch pipelining to improve the processing speed of successive images / feature maps.

[0062] FIG. 5 is a diagram illustrating batch pipelining of inference in a neural network according to some embodiments of the present invention.

[0063] In the related art, each input image / feature map may be processed one by one, so that when a first filter connected to a first layer completes processing of the first layer, it may remain idle and not process any additional information until all other layers of the neural network have completed processing.

[0064] According to some embodiments, the PIM array 10 utilizes batch processing to process one or more input images / feature maps in parallel (e.g., concurrently / simultaneously). When a filter coupled to a layer in this manner completes image / feature map processing for that layer, the filter can then process the same layer in a subsequent image / feature map.

[0065] According to some embodiments, a neural network for processing multiple images includes multiple layers that are mapped to different tiles 100 of the PIM array 10. The multiple layers may include an i-th layer (i is an integer greater than 0) and an i+1-th layer. The configurable PIM system 1 processes the i_1-th value 402 of the i-th layer for the first input image to generate an i+1_1-th value that is input to the i+1_1-th layer. The configurable PIM system 1 then processes the i_1-th value 412 of the i-th layer for the first input image to generate an output value for the next layer. According to some embodiments, while (e.g., simultaneously) processing the i+1-th value for the first image, the configurable PIM system 1 processes the i_2-th value 422 for the second input image to generate an i+1_2-th value 432. In some embodiments, the processing of the i_2-th value 422 for the second image may occur simultaneously with the processing of the i_1-th value 402 for the first input image. Processing the i_1 value 402 for the first input image may include applying the i filter 404 associated with the i layer to the i_1 value 402 of the i layer to generate the i+1 value for the i+1 layer. Also, processing the i_2 value for the second input image may include applying the i filter 404 associated with the i layer to the i_2 value to generate the i+1_2 value for the i+1 layer. That is, the same i filter 404 may be used to process the i_1 and i_2 values ​​in a time-staggered manner. However, embodiments of the present invention are not limited in this respect, and in some embodiments, a filter similar to the i filter 404 may be used to process the i_2 value, but with the same size and stride as the i filter 404.

[0066] According to some embodiments, at least one first PIM sub-array 110 may perform a filtering operation of the i-th filter 404, and at least one second PIM sub-array 110 may perform a filtering operation of the i+1-th filter of the i+1-th layer of the neural network. The controller 20 may provide the i_1-th value of the i-th layer corresponding to the first input image to at least one first PIM sub-array 110 to generate the i+1_1-th value of the i+1-th layer. The controller 20 may provide the i+1_1-th value of the i+1-th layer to at least one second PIM sub-array 110 to generate an output value associated with the first input image. While providing the i+1-th value corresponding to the first image, the controller 20 may provide the i_2-th value of the i-th layer corresponding to the second input image to at least one first PIM sub-array 110 to generate the i+1_2-th value.

[0067] As shown in FIG. 5, the i_1 values ​​402 corresponding to a first image can generate a two-dimensional matrix 400, and the i_2 values ​​422 corresponding to a second image can generate a two-dimensional matrix 420. In some embodiments, the i_1 values ​​402 include pixel values ​​of the first image (or a rectangular block of the first image), and the i_2 values ​​422 include pixel values ​​of the second image (or a rectangular block of the second image). The first and second images can be of the same size / order. In some embodiments, the i_1 values ​​402 include values ​​of a first feature map generated by a previous layer of the neural network, and the i_2 values ​​422 include values ​​of a second feature map generated by a previous layer of the neural network. The first and second feature maps correspond to (e.g., are generated from) the first and second images, respectively. The i th filter 404 may be a sliding convolution filter in the form of a p×q matrix, where p and q are integers greater than 0 (for ease of illustration in FIG. 5, the i th filter 404 is shown as a 2×2 matrix).

[0068] According to some embodiments, the processing of the i_2th value for the second input image begins a time offset after the processing of the i_1th value for the first input image begins. The time offset may be greater than or equal to a number of clock cycles corresponding to a single stride of the ith filter. For example, in the example of FIG. 5 where the stride of filter 404 is 1 corresponding to a single clock cycle, the time offset between the processing of the same layer for the first and second images may be at least one clock cycle.

[0069] According to some embodiments, the filters 404 operating on the first and second images may be copies of each other, but may be implemented in hardware via separate PIM sub-arrays 110 .

[0070] According to some embodiments, the number of images processed in parallel by the configurable PIM system 1 may be increased to improve processing time (e.g., improve image recognition time). In some embodiments, the number of images that can be batched by the configurable PIM system 1 may be limited to the number of clock cycles it takes to completely process a single image. For example, if a single image takes 100 clock cycles to process, then no more than 100 images may be batched by the configurable PIM system 1.

[0071] According to some embodiments, inter-layer and intra-layer pipelining (e.g., as described with reference to Figures 3a-3c and 4a-4c) can be used in conjunction with batch pipelining of multiple images to further improve performance.

[0072] The neural network referred to in the present invention according to some embodiments may be a convolutional neural network (ConvNet / CNN) that can be applied to an input image / video and can assign importance (e.g., learnable weights and biases) to various aspects / objects of the image / video and differentiate them from others. However, the embodiments of the present application are not limited thereto. For example, the neural network may be a recurrent neural network (RNN), a multilayer perceptron (MLP), etc.

[0073] As described herein, the pipelining scheme of the configurable PIM system according to some embodiments of the present invention provides significant (e.g., orders of magnitude) power performance improvements over other digital or PIM-based inference accelerators in the related art, and the pipelining scheme can provide low latency, stall-free operation of neural networks in hardware.

[0074] As one skilled in the art will appreciate, the operations performed by the controller 20 may be performed by a processor. Memory adjacent to the processor may contain instructions that, when executed, cause the processor to perform the operations of the controller.

[0075] In the present application, terms such as "first", "second", "third" and the like are used to describe various components, configurations, regions, layers and / or sections, but it is understood that such components, configurations, regions, layers and / or sections should not be limited by these terms. The terms are used to distinguish one component, configuration, region, layer or section from another component, configuration, region, layer or section. Thus, a first component, first element, first region, first layer or second section described below could be a second component, second configuration, second region, second layer or second section without departing from the scope of the inventive concept.

[0076] The terms used herein are intended to describe particular embodiments and are not intended to limit the inventive concepts. As used herein, the singular forms "a" and "an" include the plural unless the context clearly dictates otherwise. The terms "comprises" and / or "comprising" when used herein specify the presence of stated features, integers, steps, acts, components, and / or configurations, but do not exclude the presence of one or more other features, integers, components, and configurations. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. The use of phrases such as "at least one" when listing a list of components modifies the entire list of components, and not the individual elements of the entire list. Additionally, the use of "may" when describing embodiments of the inventive concepts means "one or more embodiments of the inventive concepts." Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0077] As used herein, the terms "use" and "using" may be synonymous with the terms "utilize" and "utilize," respectively.

[0078] The configurable PIM system and / or any associated devices or components according to the embodiments of the present invention described herein may be implemented utilizing any suitable combination of suitable hardware, firmware (e.g., ASIC), software, firmware, and hardware. For example, the various components of the PIM system may be formed on a single integrated circuit (IC) chip or separate IC chips. Also, the various components of the PIM system may be formed on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or on the same substrate. Also, the various components of the PIM system may be processes or threads executed by one or more processors, and may execute computer program instructions in one or more computing devices and interact with other system components to perform the various functions described herein. The computer program instructions are stored in a memory that may be embodied in the computing device using standard memory devices such as, for example, a random access memory (RAM). The computer program instructions may also be stored on a non-volatile computer readable medium such as a CD-ROM, a flash drive, or the like. Additionally, those skilled in the art will recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the scope of the exemplary embodiments of the present application.

[0079] Although the present application has been described in detail with particular reference to the exemplary embodiments, the embodiments set forth herein are not intended to be exhaustive or to limit the scope of the disclosure to the precise forms disclosed. Those skilled in the art to which the present disclosure pertains will recognize that modifications and variations in the described structure and methods of assembly and operation may be made without significantly departing from the principles of the present disclosure, as set forth in the following claims and their equivalents. [Explanation of symbols]

[0080] 1. PIM System 20 Controller 30 Memory 100 tiles 110 PIM Sub-array 112 bit cell 114 Digital to Analog Converter 116 Sample and Hold Circuit 120 Input Registers 140 Shift and Add Circuit 150 Output Register

Claims

1. A method for pipeline inference of a neural network including a plurality of layers including an i-th layer, an i+1-th layer, and an i+2-th layer, comprising: a controller processes a first set of the i-th values ​​of the i-th layer through an i-th filter to generate an i+1-th value for the i+1-th layer; determining an amount of the i+1 value that is determined to be sufficient for processing by the i+1 filter based on the controller generating an i+1 value corresponding to each unit operated by the i+1 filter associated with the i+1 layer; the controller, in response to the determination, simultaneously processing the i+1 value through the i+1 filter to generate a result value for the i+2 layer while processing a second set of the i values ​​for the i layer through the i filter; The method, wherein i is an integer greater than 0.

2. The method of claim 1 , wherein the first set of i values ​​comprises pixel values ​​of an image.

3. The method of claim 1 or 2, wherein the first set of i values ​​comprises values ​​of a feature map generated by a previous layer of the neural network.

4. Processing the first set of the i values ​​in the i th layer includes:

4. The method of claim 1, further comprising applying an ith filter associated with the ith layer to the first set of the ith values ​​of the ith layer to generate the i+1 value for the i+1 layer.

5. The i-th filter is a sliding convolutional filter having a p×q matrix form, The method of claim 4 , wherein p and q are integers of 0 or greater.

6. Applying the i-th filter 5. The method of claim 4, comprising performing a matrix multiplication between the i-th filter and one of the first set of i-th values ​​that overlaps the i-th filter.

7. Determining the amount of the i+1 value that is deemed sufficient for processing comprises:

7. The method of claim 1, further comprising: a clock cycle determined by at least one of a size of the i-th filter associated with the i+1 layer, a size of the i+1-th filter associated with the i+1 layer, a stride of the i+1 filter, and size of an image and a characteristic map corresponding to the first and second set of the i values ​​of the i-th layer.

8. Processing the first set of the i values ​​in the i th layer includes: applying an i_1 filter associated with the i layer to a first portion of the first set of i values ​​to generate a first region of the i+1 values ​​for the i+1 layer; 8. The method of claim 1, comprising simultaneously applying the i_1 filter and applying an i_2 filter associated with the i layer to a second portion of the first set of i values ​​to generate a second range of the i+1 values ​​for the i+1 layer.

9. The method of claim 8 , wherein the first and second portions of the first set of i values ​​have overlapping values.

10. The method of claim 8 , wherein the first and second portions of the first set of i values ​​have non-overlapping values.

11. The method of claim 8 , wherein the i_1 filter and the i_2 filter are sliding convolution filters having the same stride and the same dimension.

12. The method of claim 8 , wherein the i_1 filter and the i_2 filter have the same weighting value.

13. In a system for pipelined inference of a neural network including a plurality of layers including an i-th layer, an i+1-th layer, and an i+2-th layer, A processor; a processor memory adjacent to said processor, said processor memory storing instructions that, when executed by said processor, cause said processor to operate; The instructions include instructions for the processor to: processing a first set of i values ​​of the i layer through an i filter to generate an i+1 value for the i+1 layer; determining an amount of the i+1 value that is determined to be sufficient for processing by the i+1 filter based on generating an i+1 value corresponding to each unit operated by the i+1 filter associated with the i+1 layer; and responsive to said determining, while simultaneously processing a second set of said i values ​​for said i layer through said i filter, processing said i+1 values ​​through said i+1 filter to generate result values ​​for said i+2 layer; The system, wherein i is an integer greater than 0.

14. Processing the first set of the i values ​​in the i th layer includes:

14. The system of claim 13, comprising applying an ith filter associated with the ith layer to the first set of the ith values ​​for the ith layer to generate the i+1 value for the i+1 layer.

15. Determining the amount of the i+1 value that is deemed sufficient for processing comprises:

15. The system of claim 13 or 14, wherein the clock cycle is determined by at least one of a size of the i th filter associated with the i+1 layer, a size of the i+1 th filter associated with the i+1 layer, a stride of the i+1 filter, and based on the size of the image and characteristic map corresponding to the first and second sets of the i values ​​of the i layer.

16. Processing the first set of the i values ​​in the i th layer includes: applying an i_1 filter associated with the i layer to a first portion of the first set of i values ​​to generate a first region of the i+1 values ​​for the i+1 layer; 16. The system of claim 13, further comprising simultaneously applying the i_1 filter an i_2 filter associated with the i layer to a second portion of the first set of i values ​​to generate a second range of the i+1 values ​​for the i+1 layer.

17. In a configurable processing-in-memory (PIM) system for implementing neural networks, At least one first PIM subarray performing a filtering operation of an i-th filter in an i-th layer of the neural network; At least one second PIM subarray performing a filtering operation of the i+1th filter in the i+1th layer of the neural network; a controller for controlling the first and second PIM subarrays; The controller: providing a first set of i th values ​​for the i layer to the at least one first PIM subarray to generate an i+1 th value for the i+1 layer; determining an amount of the i+1 values ​​determined to be sufficient for processing by the at least one second PIM subarray based on generating the i+1 values ​​corresponding to each unit operated by the at least one second PIM subarray; and in response to the determining, providing the i+1 value to the at least one second PIM subarray while simultaneously providing a second set of the i values ​​of the i layer being processed to the at least one first PIM subarray to generate result values ​​for a subsequent layer of the neural network. A PIM system, wherein i is an integer greater than 0.

18. The PIM subarrays of the first and second PIM subarrays include:

20. The system of claim 17, comprising a plurality of cells storing a plurality of weighting values ​​corresponding to respective ones of the i filter and the i+1 filter.

19. the neural network includes a convolutional neural network; 19. The system of claim 17 or 18, wherein the i-th filter is a sliding convolution filter.

Images