A batch-size pipelined PIM accelerator for multi-image vision inference

Abstract

To provide a method of pipelining inference of a neural network which includes an i-th layer and an (i+1)-th layer.SOLUTION: The method includes an i-th layer (i being an integer greater than zero) and an (i+1)-th layer, and includes: processing, for a first input image, an i_1-th value of the i-th layer to generate an (i+1)-th value of the (i+1)-th layer; processing, for the first input image, an (i+1_1)-th value of the (i+1)-th layer to generate an output value; and, concurrently with processing the first (i+1) value for the first input image, processing an i_2-th value of the i-th layer for a second input image to generate an (i+1_2)-th value, thereby generating and processing the i_2-th value of the i-th layer for the second input image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] Some embodiments of the present disclosure relate generally to machine learning. [Background technology]

[0002] The explosive growth of big data-based Machine Learning (ML) applications along with the prospect of a slowdown in Moore's Law is prompting research into application-specific hardware fabrics. With a focus on putting computing inside the memory bitcells, Processing In Memory (PIM) has been proposed to accelerate machine learning inference applications. Machine learning applications and networks must be efficiently mapped onto the underlying hardware fabric to extract the best power performance.

[0003] The above information published in the Background section is intended to enhance the understanding of the present disclosure and may therefore contain information that does not constitute prior art known to one of ordinary skill in the art. Summary of the Invention [Problem to be solved by the invention]

[0004] The technical problem to be solved by the present invention is to provide a new pipelining scheme that utilizes higher area to implement inter-layer pipelining and thus has significant power-performance advantages over other digital or PIM-based inference accelerators.

[0005] The technical problem to be solved by the present invention is to provide a network topology with no stall or low latency operation on hardware and to obtain a large gain in delay time by not generating overhead. [Means for solving the problem]

[0006] Embodiments of the present disclosure relate to systems and methods for mapping machine learning (ML) applications to accelerator-based Processing In Memory (PIM) in a pipelined manner. According to some embodiments, pipelining may include inter-layer pipelining, intra-layer pipelining, and / or a combination of the two. Additionally, the pipelining scheme of various embodiments allows for generating multiple outputs in one clock cycle for multiple images simultaneously.

[0007] Some embodiments of the present disclosure are directed to a configurable hardware accelerator based on mixed-signal PIM, where a machine learning problem is mapped to PIM sub-arrays in a pipelined manner.

[0008] Some embodiments of the present disclosure are directed to a configurable hardware accelerator based on mixed-signal PIM that receives multiple images as input for recognition. Input activations for the multiple images are received in a pipelined manner to the accelerator.

[0009] A method of pipelining inference of a neural network including multiple layers, including an i-th layer (i is an integer greater than 0) and an i+1-th layer, according to some embodiments of the present disclosure, includes processing the i_1-th value of the i-th layer on a first input image to generate an i+1_1-th value of the i+1-th layer, processing the i+1_1 value of the i+1-th layer on the first input image to generate an output value, and processing the i+1_1 value on the first input image while simultaneously processing the i_2 value of the i-th layer on a second input image to generate and process an i+1_2 value.

[0010] In some embodiments of the present disclosure, the second input image is processed according to the i_2 value while the first input image is processed according to the i_1 value.

[0011] In some embodiments of the present disclosure, the i_1 value comprises a pixel value of the first input image and the i_2 value comprises a pixel value of the second input image.

[0012] In some embodiments of the present disclosure, a first feature map generated by a previous layer of the neural network includes the i_1 value corresponding to the first input image, and a second feature map generated by a previous layer of the neural network includes the i_2 value corresponding to the second input image.

[0013] In some embodiments of the present disclosure, the method includes processing the i_1 value of the i layer on the first input image and applying the i layer associated with an i filter to the i_1 value of the i layer to generate the i+1_1 value for the i+1 layer.

[0014] In some embodiments of the present disclosure, the second input image includes processing the i_2 value of the i layer and applying the i layer associated with the i filter to the i_2 value of the i layer to generate the i+1_2 value for the i+1 layer.

[0015] In some embodiments of the present disclosure, the i filter is a sliding convolution filter of p×q matrix form, where p and q are integers greater than zero.

[0016] In some embodiments of the present disclosure, the method includes applying the i-th filter and performing a matrix multiplication operation between the i-th filter and one of the values ​​of the i_1-th filter that overlaps with the i-th filter.

[0017] In some embodiments of the present disclosure, processing for the second input image includes starting a time offset for the i_1 value of the i layer for the first input image after starting processing of the i_2 value of the i layer, the time offset being greater than or equal to a number of clock cycles corresponding to a single stride of the i filter.

[0018] According to some embodiments of the present disclosure, a system for pipelining inference of a neural network including a plurality of layers including an i-th layer (i is an integer greater than 0), an i+1-th layer, and an i+2-th layer includes a processor and a processor memory necessary for operation of the processor that stores commands that, when executed by the processor, cause the processor to process the i_1-th value of the i-th layer against a first input image to generate the i+1_1-th value of the i+1-th layer, process the i+1_1-th value of the i+1-th layer against the first input image to generate an output value, and simultaneously process the i+1-th value against the first input image, process the i_2-th value of the i-th layer against a second input image to generate the i+1_2 value.

[0019] In some embodiments of the present disclosure, the second input image is processed according to the i_2 value while the first input image is processed according to the i_1 value.

[0020] In some embodiments of the present disclosure, the i_1 value comprises a pixel value of the first input image and the i_2 value comprises a pixel value of the second input image.

[0021] In some embodiments of the present disclosure, a first feature map generated by a previous layer of the neural network includes the i_1 value corresponding to the first input image, and a second feature map generated by a previous layer of the neural network includes the i_2 value corresponding to the second input image.

[0022] In some embodiments of the present disclosure, the method includes processing the i_1 value of the i layer on the first input image and applying an i filter to the i layer associated with the i filter to the i_1 value of the i layer to generate the i+1_1 value for the i+1 layer.

[0023] In some embodiments of the present disclosure, the i_2 value of the i layer is processed on the second input image, and the i_2 value of the i layer is applied to the i layer associated with an i filter to generate the i+1_2 value for the i+1 layer.

[0024] In some embodiments of the present disclosure, the i filter is a sliding convolution filter of p×q matrix form, where p and q are integers greater than zero.

[0025] In some embodiments of the present disclosure, the method includes applying the i-th filter and performing a matrix multiplication operation between the i-th filter and one of the values ​​of the i_1-th filter that overlaps with the i-th filter.

[0026] In some embodiments of the present disclosure, processing for the second input image includes starting a time offset for the i_1 value of the i layer for the first input image after the i_2 value of the i layer begins processing, the time offset being greater than or equal to a number of clock cycles corresponding to a single stride of the i filter.

[0027] In some embodiments of the present disclosure, a configurable Processing In Memory (PIM) system configured to embody a neural network includes at least one first PIM subarray performing a filtering operation of an i-th filter in an i-th layer (i is an integer greater than 0) of the neural network; at least one second PIM subarray performing a filtering operation of an i+1-th filter in an i+1-th layer of the neural network; and a controller configured to control at least one PIM subarray, wherein the controller is configured to: provide an i_1 value of the i-th layer to at least one first PIM subarray, which generates an i+1_1 value corresponding to a first input image; provide the i_1 value of the i+1 layer to at least one second PIM subarray to generate an output value associated with the first input image; and provide the first input image corresponding to the i+1_1 value while providing an i_2 value to at least one first PIM subarray to the i-th layer, which generates an i+1_2 value corresponding to a second input image.

[0028] In some embodiments of the present disclosure, a PIM subarray of at least one of the first and second PIM subarrays includes a number of bit cells storing a number of weight values ​​corresponding to the i filter or the i+1 filter, respectively. [Brief description of the drawings]

[0029] The drawings accompanying the specification illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Figure 1] FIG. 1 is a schematic diagram illustrating a configurable PIM system according to some embodiments of the present disclosure. [Figure 2a] FIG. 1 illustrates a configurable PIM system tile according to some embodiments of the present disclosure. [Figure 2b] FIG. 1 illustrates a PIM sub-array of the tile according to some embodiments of the present disclosure. [Figure 3a] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Figure 3b] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Figure 3c] FIG. 2 illustrates inter-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Figure 4a] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Figure 4b] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Figure 4c] FIG. 2 illustrates a combination of inter-layer and intra-layer pipelining of inference in a neural network according to some embodiments of the present disclosure. [Diagram 5] FIG. 2 illustrates batch pipelining of inference in a neural network according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] The detailed description set forth below is provided by the present disclosure as a description of embodiments of systems and methods for accelerating pipeline machine learning (ML) and is not intended to represent the only manner in which the present disclosure may be constructed or utilized. The description describes features of the present disclosure in conjunction with the illustrated embodiments. However, it is apparent that the same or equivalent functions and structures may be achieved by different embodiments that may be encompassed within the scope of the present disclosure. As elsewhere in the present disclosure, like reference numerals are intended to represent like components or features.

[0031] Classes of the present disclosure relate to mapping machine learning applications to a Processing In Memory (PIM)-based accelerator in a pipelined manner. Pipelining can include inter-layer pipelining, intra-layer pipelining, or a combination of the two classes. The PIM-based system is configurable, and the pipelining scheme can be used for specific applications and mapped to the PIM-based system on a case-by-case basis to increase (e.g., maximize) the power performance of each application. According to some embodiments, the pipelining scheme of the configurable PIM-based system can provide a significant (e.g., tens of times) power performance improvement over other digital or PIM-based inference accelerators of the related art. Additionally, the pipelining scheme can provide stall-free or low latency operation in a network topology in hardware. FIG. 1 is a schematic diagram illustrating a configurable PIM system according to some embodiments of the present disclosure.

[0032] 1, once trained, the configurable PIM system 1 infers input data to generate output data, such output data may be predicted based on the input data. According to some embodiments, the configurable PIM system 1 includes a PIM array 10 including multiple tiles 100 for performing inference operations, a controller 20 for controlling the operation of the PIM array 10, and a memory (30, e.g., on-logic-die memory) for storing result values ​​or intermediate results of each tile 100 of the configurable PIM system 1. In some embodiments, the memory 30 may be embedded magneto-resistive random access memory (eMRAM), static random access memory (SRAM), and / or the like.

[0033] Figure 2a is a diagram illustrating a tile 100 of a configurable PIM system 1 according to some embodiments of the present disclosure. Figure 2b is a diagram illustrating a PIM sub-array 110 of the tile 100 according to some embodiments of the present disclosure.

[0034] 2a, a tile 100 of a PIM array 10 according to some embodiments includes a plurality of PIM subarrays 110 in a matrix format, an input register 120 for receiving and storing an input signal Vin (e.g., an input voltage signal, also referred to as an input activation) that corresponds to input data and providing the input signal stored in an appropriate one of the PIM subarrays 110, an analog-to-digital converter (ADC, 130) for converting an analog output of the PIM subarray 110 to a digital signal (e.g., a binary signal), a shift and add circuit (140) for storing and adding the output of the analog-to-digital converter (ADC, 130) in one clock cycle, and an output register 150 for storing the output of the shift and add circuit 140 before forwarding the result to memory 30 for storage. In some embodiments, a controller 20 can control the operation of the input and output registers (120 and 150) and can determine which PIM subarray 110 to forward the data stored in the input register 120 to.

[0035] Referring to FIG. 2b, each PIM subarray 110 according to some embodiments includes a plurality of bit cells 112, which may be analog or digital (single-bit or multi-bit). In some embodiments, the bit cells 112 may be non-volatile synaptic weight bit cells with two or three terminals. The bit cells 112 may be resistive random access memory (RRAM) that may act as analog or digital memory. However, the present embodiments are not limited thereto, and in some embodiments, the bit cells 112 may be conductive bridging random access memory (CBRAM), phase-change memory (PCM), ferroelectric field-effect transistor (FerroFET), spin-transfer torque (STT) memory, etc. The bit cells 112 may also include multiple units of memory cells and may have three or more terminals. To select individual bit cells 112, one or more diodes or field effect transistors (FETs) may be attached in series with the bit cells 112. The PIM sub-array 110 may further include peripheral circuitry such as a digital-to-analog converter (DAC, 114) that converts a digital input to an analog voltage signal and provides it to one or more bit cells 112, and a sample and hold circuit (116) that stores the output of the bit cells 112 before transmitting it to subsequent blocks.

[0036] In some embodiments, the PIM subarray 110 can operate as a filter (eg, a convolution filter) with each bit cell 112 storing a learnable weight (w1, w2, w3, etc.) of the filter.

[0037] 2a-b, a machine learning system may be described by a network such as a convolution neural network (CNN), a fully connected (FC) neural network, and / or a recurrent neural network (RNN) having multiple interconnected layers. Such interconnected layers may be described by a weight matrix that is mapped to and stored in the bit cells 112.

[0038] In some embodiments, a set of PIM sub-arrays 110 is assigned to each layer of the neural network based on the network topology.

[0039] The number of rows of the PIM subarray 110 in the tile 100 may be determined by the filter size of the interconnected layer (e.g., a layer of a convolution neural network (CNN)). The number of columns of the PIM subarray 110 in the tile 100 may be determined by the number of filters in that layer, the number of bits mapped per bit cell 112, and the precision of the filter weights. In some embodiments, each row of the PIM subarray 110 represents a filter in a convolution neural network (CNN). Input activations from a previous layer may be fed together (e.g., simultaneously) in one clock cycle as row inputs for a row of the PIM subarray 110. In some embodiments, the rows and columns of the tile 100 may be shifted relative to one another as long as logic in the controller 20 is changed.

[0040] In some embodiments, the PIM subarray 110 outputs a conductive weighted output current I OUT(which may represent synaptic neurons), and the output currents of the PIM subarrays 110 along a row are summed together (e.g., by a common electrical connection coupled to the outputs of the PIM subarrays 110). Thus, a row of PIM subarrays 110 may generate I OUT1 Or I OUTn The summed output currents may be converted to a digital signal (e.g., binary data) by an ADC (analog-to-digital converter, 130). Each summed output may represent a calculated partial sum, which is shifted by a shift and add circuit (140) and added with the partial sum of the next input activity. An output register 150 stores the calculated partial sum. Once the controller 20 determines that a final output has been generated, it stores the final output (back) in the memory 30, which may be used as an input for the next layer. The method and timing used to process this output data in the next layer may determine the pipelining scheme being implemented.

[0041] In some embodiments, the mapping of network layers to the PIM subarrays 110 is done in a pipelined manner, which may result in stall-free (or low stall) operation. In some embodiments, rather than waiting for a neural network layer to complete processing before moving on to the next layer, the controller 20 starts processing the next layer of the neural network once sufficient output has been generated from the current layer. This is referred to as inter-layer pipelining (also known as inter-layer parallelism).

[0042] 3a-3c are diagrams illustrating inter-layer pipelining of inference in a neural network according to some embodiments of the present disclosure.

[0043] 3a-3c, a set of values ​​for the i-th layer may form a two-dimensional matrix 300, where i is an integer greater than 0. In some embodiments, the two-dimensional matrix 300 represents an image, where each element 302 of the matrix 300 represents a pixel value (e.g., color intensity). The two-dimensional matrix 300 represents a feature map, where each element 302 represents an output value of a previous layer (i.e., the i-1-th layer). For ease of explanation, the two-dimensional matrix 300 in FIG. 3a-3c is a 6×6 matrix, but as will be appreciated by those skilled in the art, embodiments are not limited thereto, and the matrix 300 may have any suitable size, represented as n×m, where n and m are integers greater than 1.

[0044] The i-th filter 304 (also referred to as a kernel) is a sliding convolutional filter that operates on the set of values ​​of the i-th layer to generate values ​​for the next layer (i.e., the i+1-th layer). The filter 304 may be represented as a two-dimensional matrix of size p×q, where p and q are integers greater than 0 and less than or equal to n and m, respectively. For ease of illustration, the i-th filter 304 is represented as a 2×2 filter. Each component of the convolutional filter may be a learnable weight.

[0045] The i-th filter 304 shifts / slides / moves across the set of values ​​of the i-th layer by a stride length until the entire set of values ​​of the i-th layer has been passed. At each shift, the i-th filter 304 performs a matrix multiplication operation between the i-th filter 304 and the matrix portion of the i-th layer value on which the filter 304 is currently operating. Although the embodiment of Figures 3a-3c illustrates a stride length of 1, the stride length may be 2, 3, or any suitable value. The output of the convolution operation that produces the set of values ​​of the next layer (i.e., layer i+1) is referred to as the convolved characteristic output. Such output may fill the matrix 310 associated with the i+1-th layer.

[0046] In the related art, layers of a neural network must be completely processed before processing the values ​​of subsequent layers, which can be slow.

[0047] However, according to some embodiments, the controller 20 monitors the output values ​​generated by the i-th filter 304, and when it determines that there are a sufficient amount of values ​​available for processing in the next layer (i.e., the i+1-th layer), the controller 20 processes the output values ​​available in the i+1-th layer through the i+1-th filter 314. Thus, the controller 20 may apply the i+1-th filter 314 to the available values ​​of the i+1-th layer while simultaneously applying the i-th filter 304 to the values ​​of the i-th layer. Such inter-layer pipelining may be performed for any number or all of the neural network. That is, convolution operations may be performed simultaneously (e.g., may be performed simultaneously in two or more layers of a neural network). This may significantly improve the inference speed of the neural network.

[0048] According to some embodiments, at least one first PIM sub-array 110 performs the filtering operation of the i-th filter 304, and at least one second PIM sub-array 110 performs the filtering operation of the i+1-th filter 314. The controller 20 may provide a first set of the i values ​​of the i-th layer to at least one PIM sub-array 110 to generate the i+1 values ​​for the i+1-th layer, and when it is determined that the amount of the i+1 values ​​is sufficient for processing at least one second PIM sub-array 110, the controller 20 may provide a second set of the i values ​​for the processing to the first PIM sub-array 110 while providing the i+1 values ​​to the second PIM sub-array 110 to generate output values ​​for a subsequent layer of the neural network. In some embodiments, it is determined that the amount of values ​​available for the filtering operation is sufficient when there is data for each device operating with the corresponding filter in the corresponding layer.

[0049] In some embodiments, the time (e.g., clock cycles) at which processing of the next layer (i.e., layer i+1) can begin may depend on the size of the i filter associated with the i layer, the size of the i+1 filter associated with the i+1 layer, the stride of the i+1 filter, and the size of the image or feature map corresponding to the i layer values. In the example of Figures 3a-3c, each of the i and i+1 filters 304, 314 has a size of 2x2, the i filter 304 has a stride length of 1, and the i+1 filter 314 has a stride length of 2, and the controller 20 begins processing the i layer and 7 cycles later begins processing the i+1 layer. As shown in Figure 3c, the i+1 filter 314 runs out of values ​​available in the feature map 310 to perform additional operations. According to some embodiments, each convolution operation may be performed in one clock cycle.

[0050] The processing speedup from inter-layer pipelining according to some embodiments of the present disclosure can be further improved by additionally using intra-layer pipelining (also called intra-layer parallelism) to generate more than one output value from a layer per cycle.

[0051] 4a-4c illustrate a combination of layers of inference and intra-layer pipelining in a neural network according to some embodiments of the present disclosure.

[0052] With reference to Figures 4a-4c, in some embodiments, one or more filters at each layer of a neural network operate together (eg, simultaneously) to generate one or more output values ​​at a time.

[0053] In some embodiments, at a given time, the controller 20 applies an i_1th filter (e.g., an i_1th sliding filter, 304) associated with the ith layer to a first portion / block of the ith layer values ​​to generate a first output value (e.g., a first region of i+1th values) and (e.g., simultaneously) applies an i_2th filter (e.g., an i_2th sliding filter, 305) associated with the ith layer to a second portion / block of the ith layer values ​​to generate a second output value (e.g., a second region of i+1th values).

[0054] According to some embodiments, the i_1th and i_2th filters are identical (e.g., contain the same weight map / weights), but offset in position by the stride length of the first filter 304. Thus, the i_2th filter 305 effectively performs the same operation as the i_1th filter 304 did in the next clock cycle, but in the same clock cycle as the i_1th filter 304. As a result, in one clock cycle, the PIM array 10 can generate two (or more) values ​​for the next layer. Here, each layer can be subdivided and mapped to other tiles using one or more copies of the same weight matrix. For example, the i_1th filter 304 can be implemented in one tile 100, and the i_2th filter 305 can be implemented in another tile 100, such that the weight matrix for the two tiles 100 is identical.

[0055] In some embodiments, the number of parallel operations performed by a filter in each layer may be equal to the number of filters that make up the combined / composite filter (e.g., 303) in that layer. Here, the composite stride of the composite filter may be equal to the number of filters that make up the composite filter (and perform parallel operations) multiplied by the strides of the constituent filters. For example, in Figures 4a-4c, the stride length of composite filter 303, which includes two i-th filters 304, 305, each with a stride length of 1, is equal to 2. As will be appreciated by those skilled in the art, embodiments of the present invention are not limited to two parallel operations per layer and may be extended to include any suitable number of parallel operations.

[0056] As shown in Figures 4a-4c, the layer and intra-layer pipelining (this combination is also called "ultra-pipelining") controller 20 can start the i+1 filter processing after 4 clock cycles and continue to process the i+1 layer in the next clock cycle (t=5) without delay. This shows an improvement when compared with the inter-layer pipelining scheme of the embodiment of Figures 3a-3c, where the controller 20 can start the i+1 filter processing only after 7 clock cycles and the amount of values ​​available in the i+1 layer in the next clock cycle (t=8) is insufficient to perform the next filtering operation of the i+1 filter.

[0057] Although inter-layer pipelining / parallelism has been described for the i-th layer, according to some embodiments, PIM array 10 may use inter-layer pipelining / parallelism at one or more layers (e.g., at all layers) of the neural network.

[0058] In addition to using inter-layer pipelining to improve (e.g., increase) the processing speed of a single image or feature map, embodiments of the present disclosure utilize batch pipelining to improve the processing speed of successive images / feature maps.

[0059] FIG. 5 is a diagram illustrating batch pipelining of inference in a neural network according to some embodiments of the present disclosure.

[0060] In the related art, each input image / feature map can be processed one by one, so that once a first filter associated with a first layer has completed its first layer processing, it remains idle and does not need to process any additional information until all other layers of the neural network have completed processing.

[0061] According to some embodiments, PIM array 10 utilizes batch processing to process one or more input images / feature maps in parallel (e.g., together / simultaneously) such that when a filter associated with a layer completes image / feature map processing for that layer, the filter can process the same layer of a subsequent image / feature map.

[0062] According to some embodiments, the neural network for processing multiple images collectively includes multiple layers that are mapped to different tiles 100 of the PIM array 10. The multiple layers may include an i-th layer (i is an integer greater than 0) and an i+1-th layer. The configurable PIM system 1 processes the i_1-th value 402 of the i-th layer for a first input image to generate an i+1_1-th value 412 that is provided as an input for the i+1-th layer. The configurable PIM system 1 processes the i+1-th value 412 of the i+1-th layer for the first input image to generate an output value for a subsequent layer. According to some embodiments, while (e.g., simultaneously) processing the i+1-th value of the first input image, the configurable PIM system 1 processes the i_2-th value 422 of the i-th layer for a second input image to generate an i+1_2 value 432. In some embodiments, the processing of the i_2 value 422 for the second input image may occur simultaneously with the processing of the i_1 value 422 for the first input image. Processing the i_1 value 402 for the first input image may include applying an i_1 filter 404 associated with an i-th layer to the i_1 value 402 of the i-th layer to generate the i+1_2 value for the i+1-th layer. That is, the same i_1 filter 404 may be used to process the i_1 and i_2 values ​​in a time-staggered manner. However, embodiments of the present disclosure are not limited in this respect, and in some embodiments, a filter similar to the i_1 filter 404 but having the same size and stride as the i_1 filter 404 may be used to process the i_2 value.

[0063] According to some embodiments, at least one first PIM sub-array 110 may perform a filtering operation of the i-th filter 404, and at least one second PIM sub-array 110 may perform a filtering operation of the i+1-th filter of the i+1-th layer of the neural network. The controller 20 may provide the i_1-th value of the i-th layer corresponding to the first input image to at least one first PIM sub-array 110 to generate the i+1_1 value of the i+1-th layer. The controller 20 may provide the i+1_1 value of the i+1-th layer to at least one second PIM sub-array 110 to generate an output value associated with the first input image. Simultaneously with providing the i+1-th value corresponding to the first input image, the controller 20 may provide the i_2-th value of the i-th layer corresponding to the second input image to at least one first PIM sub-array 110 to generate the i+1_2 value.

[0064] As shown in FIG. 5, the i_1 values ​​402 corresponding to the first input image may generate a two-dimensional matrix 400, and the i_2 values ​​422 corresponding to the second input image may generate a two-dimensional matrix 420. In some embodiments, the i_1 values ​​402 include pixel values ​​of the first input image (or a rectangular block of the first image), and the i_2 values ​​422 include pixel values ​​of the second input image (or a rectangular block of the second image). The first input image and the second input image may be of the same size / dimension. In some embodiments, the i_1 values ​​402 include values ​​of a first feature map generated by a previous layer of a neural network, and the i_2 values ​​422 include values ​​of a second feature map generated by a previous layer of the neural network. The first and second feature maps correspond to (e.g., are generated from) the first and second input images, respectively. The i-th filter 404 may be a sliding convolution filter in the form of a p×q matrix, where p and q are integers greater than 0 (in FIG. 5, for ease of illustration, the i-th filter 404 is shown as a 2×2 matrix).

[0065] According to some embodiments, processing of the i_2th value for the second input image begins a time offset after processing of the i_1th value for the first input image begins. The time offset may be greater than or equal to a number of clock cycles corresponding to a single stride of the ith filter. For example, in the example of FIG. 5, the stride of filter 404 is 1, which corresponds to a single clock cycle, and the time offset between processing of the same layer for the first and second input images may be at least one clock cycle.

[0066] According to some embodiments, the filters 404 operating on the first and second images may be copies of each other, but may be implemented in hardware via other PIM sub-arrays 110 .

[0067] According to some embodiments, increasing the number of images processed in parallel by configurable PIM system 1 improves processing time (e.g., improves image recognition time). In some embodiments, the number of images that can be batch processed by configurable PIM system 1 may be limited to the number of clock cycles it takes to completely process a single image. For example, if it takes 100 clock cycles to process a single image, then no more than 100 images may be batch processed by configurable PIM system 1.

[0068] According to some embodiments, inter-layer and intra-layer pipelining (e.g., as described with reference to Figures 3a-3c and 4a-4c) can be used in conjunction with batch pipelining of multiple images to further improve performance.

[0069] The neural network referred to in the present disclosure according to some embodiments may be a convolutional neural network (ConvNet / CNN) that can take an input image / video and assign importance (e.g., learnable weights and biases) to various aspects / objects of the image / video to differentiate them from others. However, embodiments of the present disclosure are not limited thereto. For example, the neural network may be a recurrent neural network (RNN), a multilayer perceptron (MLP), etc.

[0070] As described herein, the pipelining scheme of the configurable PIM system according to some embodiments of the present disclosure provides significant (e.g., orders of magnitude) power performance improvements over other digital or PIM-based inference accelerators in the related art, and the pipelining scheme can provide low latency, stall-free operation of neural networks in hardware.

[0071] As will be appreciated by those of ordinary skill in the art, the operations performed by the controller 20 may be performed by a processor. Memory adjacent to the processor may contain instructions that, when executed, cause the processor to perform the operations of the controller.

[0072] Although terms such as "first", "second", "third", etc. are used in the present disclosure to describe various components, configurations, regions, layers and / or sections, it is understood that such components, configurations, regions, layers and / or sections should not be limited by these terms. The terms are used to distinguish one component, configuration, region, layer or section from another component, configuration, region, layer or section. Thus, a first component, first element, first region, first layer or first section described below could be a second component, second element, second region, second layer or second section without departing from the scope of the inventive concept.

[0073] The terms used in this disclosure are intended to describe specific embodiments and are not intended to limit the inventive concept. As used in this disclosure, the singular forms "a" and "an" include the plural unless the context clearly dictates otherwise. When used in this application, the terms "comprises," "comprises," and / or "has" specify the presence of stated features, integers, steps, operations, components, and / or configurations, but do not exclude the presence of one or more other features, integers, components, and configurations. As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the linked listed items. When used to describe a list of components, phrases such as "at least one" modify the entire list of components, but not the individual elements of the entire list. Additionally, when describing an embodiment of a concept, the use of "may" means "one or more embodiments of the inventive concept." Additionally, the term "exemplary" is intended to refer to an example or illustration.

[0074] As used in this disclosure, the terms "use", "utilize" and "utilize" may be synonymous with the terms "utilize", "utilize" and "utilize", respectively.

[0075] The configurable PIM system and / or any associated devices or components according to embodiments of the present disclosure described herein may be implemented utilizing any suitable combination of hardware, firmware (e.g., application-specific integrated circuit (ASIC)), software, firmware, and hardware. For example, the various components of the PIM system may be implemented on a single integrated circuit (IC) chip or separate IC chips. Also, the various components of the PIM system may be implemented on a flexible printed circuit film, tape carrier package (TCP), printed circuit board (PCB), or the same substrate. Also, the various components of the PIM system may be processes or threads executed by one or more processors, and may interact with other system components to execute computer program instructions in one or more computing devices and perform various functions described herein. The computer program instructions may be stored in a memory that may be implemented in a computing device using standard memory devices such as, for example, random access memory (RAM). The computer program instructions may also be stored on other computer readable media that are not transferable, such as, for example, a CD-ROM, a flash drive, etc. Additionally, those skilled in the art will recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the scope of exemplary implementations of the subject matter of the present disclosure.

[0076] Although the present disclosure has been described in detail with specific reference to exemplary embodiments, the embodiments described herein are not intended to completely limit the scope of the disclosure to the precise form disclosed. Those skilled in the art and in the field to which the present disclosure pertains will recognize that changes and substitutions in the structure and assembly and operation methods may be made without meaningfully departing from the principles and scope of the present disclosure, as set forth in the claims and their equivalents. [Explanation of symbols]

[0077] 1. PIM System 10 PIM Array 100 tiles 110 PIM Sub-array 112 bit cell 114 Digital to Analog Converter 116 Sample and Hold Circuit 120 Input Registers 140 Shift and Add Circuit 150 Output Register 304,404 i-th filter 305 i_2nd filter 314 (i+1)th Filter 402 i_1th value 412 i+1_1th value 422 i_2 value 432 i+1_2 value

Claims

1. A method of pipelining inference of a neural network including a plurality of layers, including an i-th layer (i is an integer greater than 0) and an i+1-th layer, comprising: a i_1 value of the i layer corresponding to a first input image is processed by a controller using a composite filter comprising an i_1 filter associated with the i layer and an i_2 filter associated with the i layer to generate an i+1_1 value for the i+1 layer, the stride length of the composite filter being equal to the sum of the stride length of the i_1 filter and the stride length of the i_2 filter; processing the i+1_1 value of the i+1 layer corresponding to the first input image to generate an output value; While processing the i+1_1 value corresponding to the first input image, processing the i_2-th value of the i-th layer corresponding to a second input image to generate an i+1_2-th value; method.

2. the i_2 value corresponding to the second input image is processed at the same time that the i_1 value corresponding to the first input image is processed; The method of claim 1.

3. the i_1 value comprises a pixel value of the first input image; the i_2 value comprises a pixel value of the second input image; The method according to claim 1 or 2.

4. The i_1 term corresponding to the first input image includes a first feature map generated by a previous layer of the neural network; The i_2 term corresponding to the second input image includes a second feature map generated by a previous layer of the neural network.

4. The method according to claim 1 .

5. Processing the i_1 value of the i layer corresponding to the first input image comprises: applying an i-th filter associated with the i-th layer to the i_1 value of the i-th layer to generate the i+1_1 value for the i+1 layer.

5. The method according to any one of claims 1 to 4.

6. Processing the i_2 value of the i layer corresponding to the second input image includes: applying the i filter associated with the i layer to the i_2 value of the i layer to generate the i+1_2 value for the i+1 layer. The method according to claim 5.

7. 6. The method of claim 5, wherein the i-th filter is a sliding convolution filter of p×q matrix form, where p and q are integers greater than 0.

8. Applying the i-th filter performing a matrix multiplication operation between the i filter and a set of i_1 values ​​that overlap the i filter; The method according to claim 5.

9. processing the i_2 value of the i layer corresponding to the second input image includes starting a time offset after starting processing the i_1 value of the i layer corresponding to the first input image; the time offset is greater than or equal to a number of clock cycles corresponding to a single stride of the i filter; 9. The method according to any one of claims 5 to 8.

10. In a system for pipelining inference of a neural network including a plurality of layers including an i-th layer (i is an integer greater than 0), an i+1-th layer, and an i+2-th layer, A processor; and a processor memory necessary for the operation of said processor for storing commands; The command, when executed by the processor, causes the processor to: processing, by a controller, the i_1-th value of the i-th layer corresponding to a first input image to generate the i+1-th value of the i+1-th layer using a composite filter comprising an i_1-th filter associated with the i-th layer and an i_2-th filter associated with the i-th layer; processing the i+1_1 value of the i+1 layer corresponding to the first input image to generate an output value; while processing the i+1_1 value corresponding to the first input image, processing the i_2 value of the i-th layer corresponding to a second input image to generate an i+1_2 value; The stride length of the composite filter is equal to the sum of the stride length of the i_1 filter and the stride length of the i_2 filter. system.

11. the i_2 value corresponding to the second input image is processed at the same time that the i_1 value corresponding to the first input image is processed; The system of claim 10.

12. the i_1 value comprises a pixel value of the first input image; 12. The system of claim 10 or 11, wherein the i_2 value comprises a pixel value of the second input image.

13. The method according to claim 12, wherein the i_1 term corresponding to the first input image includes a first feature map generated by a previous layer of the neural network; The i_2 term corresponding to the second input image includes a second feature map generated by a previous layer of the neural network. A system according to any one of claims 10 to 12. Processing the i_1 value of the i layer corresponding to the first input image comprises: applying an i-th filter associated with the i-th layer to the i_1 value of the i-th layer to generate the i+1_1 value for the i+1 layer. A system according to any one of claims 10 to 13. Processing the i_2 value of the i layer for the second input image includes: applying an i_2 value of the i_1 layer to an i_2 filter associated with the i_1 layer to generate the i+1_2 value for the i+1 layer. The system of claim 14.

16. 15. The system of claim 14, wherein the i-th filter is a sliding convolution filter of p x q matrix form, where p and q are integers greater than 0.

17. Applying the i-th filter performing a matrix multiplication operation between the i filter and a set of i_1 values ​​that overlap the i filter; The system of claim 14.

18. The processing of the i_2 value of the i layer corresponding to the second input image includes starting a time offset after starting processing of the i_1 value of the i layer corresponding to the first input image; the time offset is greater than or equal to a number of clock cycles corresponding to a single stride of the i filter; 18. A system according to any one of claims 14 to 17.

Images