Performing Dot Product Operations Using Memristor-Based Crossbar Arrays

Abstract

Provides a reduction in temp size across base tables. [Solution] A method, computer system, and computer program product are provided for performing matrix convolution on a multidimensional input matrix to obtain a multidimensional output matrix. The matrix convolution can include a set of dot product operations to obtain all elements of the output matrix. Each dot product operation in the set of dot product operations can include an input submatrix of an input matrix and at least one convolution matrix. The method can include providing a memristor-based crossbar array configured to implement vector-matrix multiplication. The method includes computing a subset of the set of dot product operations by storing convolution matrices that are the subset of the dot product operations within the crossbar array, and inputting an input vector that includes all distinct elements of the subset's input matrices to the crossbar array.

Description

[Technical field]

[0001] The present invention relates to the field of digital computer systems, and more particularly to a method for performing a set of matrix convolutions on a multi-dimensional input matrix to obtain a multi-dimensional output matrix using a memristor-based crossbar array. [Background technology]

[0002] Computer memory is a promising approach in the field of non-von Neumann computing paradigms, where nanoscale resistive memory devices store data in a continuous fashion to perform basic computational tasks. For example, by arranging these devices in a crossbar configuration, matrix-vector multiplication can be performed. However, improvements in the use of these crossbar configurations continue to be needed. Summary of the Invention

[0003] Various embodiments of the invention provide a method for performing a matrix convolution on a multidimensional input matrix to obtain a multidimensional output matrix using a memristor-based crossbar array and a crossbar array as described by the subject matter of the independent claims. The embodiments of the invention can be combined with each other if they are not mutually exclusive.

[0004] In one embodiment, the invention relates to a method for performing matrix convolution on a multi-dimensional input matrix to obtain a multi-dimensional output matrix. The matrix convolution can include a set of dot product operations to obtain all elements of the output matrix. Each dot product operation of the set of dot product operations can include an input sub-matrix of an input matrix and at least one convolution matrix. The method includes providing a memristor-based crossbar array configured to implement vector-matrix multiplication, computing a subset of the set of dot product operations by storing, within the crossbar array, a convolution matrix that is the subset of the dot product operations, and inputting an input vector to the crossbar array that includes all distinct elements of the subset of input matrices.

[0005] In another embodiment, the invention relates to a memristor-based crossbar array for performing a matrix convolution on a multidimensional input matrix to obtain a multidimensional output matrix. The matrix convolution can include a set of dot product operations to obtain all elements of the output matrix. Each dot product operation of the set of dot product operations can include an input sub-matrix of an input matrix and at least one convolution matrix. The crossbar array can be configured to store the convolution matrices within the crossbar array such that an input vector including all elements of different input sub-matrices can be input to the crossbar array to perform a subset of the dot product operations of the set of dot product operations.

[0006] The following embodiments of the invention are explained in more detail, by way of example, with reference to the drawings, in which: [Brief description of the drawings]

[0007] [Figure 1] Figure 1 shows a crossbar array of memristors. [Diagram 2]FIG. 2 shows a flowchart of a method for performing multiple dot product operations according to an embodiment of the present invention. [Diagram 3] FIG. 3 shows a block diagram illustrating a method for performing multiple dot product operations according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart of a method for performing at least a portion of a convolutional neural network estimation process according to an embodiment of the present invention. [Figure 5A] FIG. 5A is a block diagram illustrating a method for multiple dot product operations according to an embodiment of the present invention. [Figure 5B] FIG. 5B is a block diagram illustrating a method for multiple dot product operations according to an embodiment of the present invention. [Figure 6] FIG. 6 is a block diagram illustrating a method for multiple dot product operations according to an embodiment of the present invention. [Figure 7] FIG. 7 shows a graphical representation of one ResNet architecture, according to an embodiment of the present invention. [Figure 8] FIG. 8 shows a block diagram of a system according to an embodiment of the present invention. [Figure 9] FIG. 9 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Figure 10] FIG. 10 illustrates a set of functional abstraction layers provided by the cloud computing environment illustrated in FIG. 9 in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Descriptions of various embodiments of the present invention are presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable others skilled in the art to understand the disclosure of this specification.

[0009] A matrix-vector multiplication of a matrix W and a vector x can be realized through a memristor-based crossbar array by expressing each matrix element in terms of the conductance of the corresponding memristor-based crossbar array of the array. A multiplication of a matrix W and a vector x can be performed by inputting a voltage representing the value of the vector to the crossbar array. The resulting current represents the product of W and x. The resistive memory elements (or devices) of the crossbar array can be, for example, one of phase change memory (PCM), metal oxide resistive RAM, conductive bridge RAM, and magnetic RAM. In another embodiment, the crossbar array can be charge-based memory elements such as SRAM and flash (NOR and NAND) elements. A representation scheme of the matrix W and the conductance G of the crossbar array that allows the final product to be obtained is the following scheme:

[0010]

number

[0011] Here, G max is given by the conductance range of the crossbar array, W max is selected depending on the size of the matrix W.

[0012] Embodiments of the present invention may provide areas of efficient use of crossbar arrays. This allows for improved parallel computation of dot product operations. By providing a single vector with all of the elements of the input sub-matrices, the convolution matrix may be stored in a compact manner within the crossbar array. For example, embodiments of the present invention may be used for neural network training and estimation.

[0013] A submatrix of a multidimensional input matrix is ​​also a multidimensional matrix. For example, if the input matrix has size x m *y m *d m and the size of the sub-matrix of the input matrix is

[0014]

number

[0015] where subx in <x m And suby in <y in and d in are identical to the input matrix and its sub-matrices. The columns of a sub-matrix of an input matrix are consecutive columns of the input matrix, and the rows of a sub-matrix are consecutive rows of the input matrix. A multi-dimensional input matrix is ​​d in A submatrix can be referred to as a feature map with channels. in *suby in d in The channel matrix of the submatrix has the same element position (subx in ,suby in ) The dot product operation of the set of multiple dot product operations includes an input submatrix of an input matrix and at least one other convolution matrix. For example,in *suby in *d in The dot product operation of d in kernels, where each kernel is in *suby in It has a size of d in The kernels can be the same or different kernels.

[0016] The multidimensional output matrix is ​​x out *y out *d out The elements of the output matrix are either single values ​​or elements (X out ,Y out ,d out ) The pixels of the output matrix can be defined by d out The elements of the output matrix can be defined by the submatrix subx in *suby in *d in can be obtained by the dot product of the kernel of d in The kernel is associated with the channel of the output matrix to which the above element belongs. in kernel, i.e., to get all elements of the output matrix, we need a kernel of size (subx in ,suby in )d out *d in The kernel can be used to perform a set of dot product operations.

[0017] According to one embodiment, the computing step can include selecting a subset of dot product operations, where computing the subset of dot product operations provides elements along two dimensions of the output matrix, and where each selected subset of dot product operations includes a different input vector. The subset of dot product operations are selected such that they can be performed at once by the crossbar array. For example, the subset of dot product operations can be performed in parallel by inputting all elements of an input vector into the crossbar array at the same time.

[0018] According to one embodiment, the calculating step selects a subset of dot product operations, where calculation of the subset of dot product operations provides elements along three dimensions of the output matrix, and where each selected subset of dot product operations includes a different input vector.

[0019] According to one embodiment, training or inference of a convolutional neural network (CNN) includes layer operations that can be computed by a memristor-based crossbar array at each layer of the CNN, where a convolution of a matrix is ​​a layer operation for a given layer of the CNN.

[0020] According to one embodiment, the method can include providing a further memristor-based crossbar array, each further layer of the CNN associated with the memristor-based crossbar array, interconnecting the memristor-based crossbar array for execution in a pipelined manner, and performing computational steps for each further layer of the CNN using a respective subset of dot-product operations and the memristor-based crossbar array associated with the further layer.

[0021] According to one embodiment, the subset of dot product operations computed by each memristor-based crossbar array is selected such that the bandwidth requirements for each interconnect between the memristor-based crossbar arrays are identical.

[0022] According to one embodiment, the memristor crossbar array may include row lines and column lines intersecting the row lines. The resistive memory elements are coupled between the row lines and the column lines at intersections formed by the row and column lines. Each resistive memory element of the plurality of resistive memory elements may represent an element of the matrix, and wherein storing the convolution matrix includes, for each dot product operation of the subset of dot product operations, storing all elements of the convolution matrix included in the dot product operation in the resistive memory elements of each single column line of the crossbar array. This allows compact storage of the convolution matrix and allows use of the crossbar array for further parallel computation.

[0023] According to one embodiment, the column lines of the convolution matrix can be contiguous lines of the crossbar array, which allows compact storage of the convolution matrix and allows use of the crossbar array for further parallel computation.

[0024] According to one embodiment, a memristor crossbar array includes resistive memory elements coupled between row lines and column lines intersecting the row lines and at intersections formed by the row and column lines. The resistive memory elements of the plurality of resistive memory elements can represent elements of a matrix, where storing the convolution matrix includes storing all elements of the convolution matrix included in each dot product operation for a subset of the dot product operations within each single column line. The column lines of the convolution matrix can be consecutive lines of the crossbar array.

[0025] According to one embodiment, a memristor-based crossbar array includes resistive memory elements coupled between row lines and column lines intersecting the row lines and at intersections formed by the row and column lines. The resistive memory elements of the plurality of resistive memory elements can represent elements of a matrix, and storing the convolution matrix includes identifying groups of convolution matrices to be multiplied by the same input sub-matrix, identifying all elements of each convolution matrix of the group in the same row line, storing all elements of each convolution matrix of the group in the same row line, and repeating the identifying and storing steps for zero or more additional groups of convolution matrices of the subset of dot product operations. This embodiment allows for efficient use of the surface area of ​​the crossbar array. This allows for the execution of a maximum number of parallel dot product operations.

[0026] According to one embodiment, a memristor crossbar array includes resistive memory elements coupled between row lines and column lines intersecting the row lines and at intersections formed by the row and column lines, where the resistive memory elements of the plurality of resistive memory elements represent elements of a matrix, allowing controlled fabrication of a crossbar array highly suitable for performing dot product operations.

[0027] According to one embodiment, the method further includes training a convolutional neural network (CNN), which may be configured to perform the inputting and storing steps.

[0028] According to one embodiment, the CNN can be configured to perform a further set of dot product operations using the crossbar array by storing all the convolution matrices and repeating the input step for each of the further sets, where the set of dot product operations and the further set of dot product operations form all dot product operations required during estimation of the CNN.

[0029] According to one embodiment, the CNN can be configured to perform additional sets of dot product operations using the crossbar array by successively repeating the storing and inputting steps for each additional set.

[0030] The embodiments of the present invention are advantageous because they allow the most expensive computations involved in training or inferring a CNN. For example, the inference stage of a CNN can be the major convolution complexity. For example, the convolutional layers of a CNN include more than 90% of the total computations required. For example, training a CNN or inferring a trained CNN can include operations or computations such as dot products or convolutions in each layer of the CNN. A dot product operation can be computed through many multiplication-add operations, each of which computes the product of two operands and the addition of the result. In a CNN, the total number of dot calculations is relatively high; for example, for a 224×224 image, classification of one category labeling with 1000 classifications requires nearly 1 Gigabyte of operations using AlexNet.

[0031] An embodiment of the present invention allows the use of parallel activation computation of feature maps to provide a pipeline speedup for CNN(s) execution while maintaining the same communication bandwidth and memory requirements. In a pipeline execution of a CNN, one feature pixel across all channels can be computed per computation cycle. The feature map pixels are communicated to the next in-memory computation unit in the pipeline.

[0032] According to one embodiment, the input matrix is ​​the activation matrix of the feature map of the CNN, and the convolution matrix is ​​the kernel.

[0033] According to one embodiment, the input matrix is ​​the pixels of an image or the activation matrix of a feature map of a CNN.

[0034] FIGURE 1 shows a crossbar array of memristors (or resistive processing units (RPUs)) that provides local data storage along with voltage sequences illustrating the operation of the memristors. FIGURE 1 is a diagram of a two-dimensional (2D) crossbar array 100 that can perform, for example, matrix-vector multiplication. The crossbar array 100 is formed from a set of conductive row wires 102a...102n and conductive column wires 108a...m that cross the set of conductive row wires 102-n.

[0035] The conductive column wires may be referred to as column lines, and the conductive row wires may be referred to as row lines. The intersections between sets of row wires and sets of column wires are separated by memristors, which are themselves shown in FIG. 1 as resistive elements, each with an adjustable / updatable resistive weight or conductance, which are G ij , where i=1...n, and j=1...m, respectively. For ease of illustration, only one memristor 120 is labeled with a reference numeral in FIG. 1. FIG. 1 is an example of a memristor for illustrative purposes and not by way of limitation. For example, the intersections between a set of row wires and a set of column wires of a crossbar array can include charge-based memory elements instead of memristors.

[0036] Input voltage ν1...ν n are applied to the row wires 102a-n, respectively. Each column wire 108a-n is connected to a corresponding current I1, I2, ... I3 generated by each memristor along that particular column wire. mFor example, as shown in FIG. 1, a current I2 is generated by column wire 108b and can be expressed in Equation 1 as follows:

[0037]

number

[0038] That is, the array 100 is 1-n The matrix-vector multiplication is calculated by multiplying the value stored in the memristor by the input of the row wire in the memristor defined by: Thus, the multiplication can be performed locally in each memristor of array 100 using the memristor itself and the associated row or column wires of array 100.

[0039] The crossbar array of Figure 1 is capable of computing, for example, the multiplication of a vector x by a matrix W. ij can be mapped into the corresponding conductance of the crossbar array according to Equation 2 as follows:

[0040]

number

[0041] Here, G max is given by the conductance range of the crossbar array 100, and W max is selected depending on the size of the matrix W.

[0042] The size of the crossbar array 100 is given by the number of row lines, n, and the number of column lines, m, where the number of memristors is n×m. In one embodiment, n=m.

[0043] 2 is a flow chart of a method for performing at least a portion of a matrix convolution on a multi-dimensional input matrix. The matrix convolution can provide a multi-dimensional output matrix. For purposes of simplicity, but without limitation, the method of FIG. 2 will be described with reference to the example of FIG. 3. For example, an input matrix 321 may have dimensions x in ,y in and x in *y in 3 as having din, which specifies the number of *din elements. in ,y in ,x out *y out *We denote the output matrix 323 as having dout, which defines the number of dout elements. For ease of explanation, din and dout are chosen to be equal to 1 in the embodiment of FIG.

[0044] To obtain all elements of the output matrix 323, the matrix convolution on the input matrix 321 may include a set of dot product operations. For example, the set of dot product operations may include din*dout convolution matrices of size k*k. For example, one element of the output matrix 323 may be obtained by each dot product operation, where the result of the dot product operation may be a single column output of the crossbar array. The single column stores all the convolution matrices required to perform the dot product operation. The set of dot product operations may be decomposed into multiple subsets of dot product operations, so that each subset of dot product operations may be performed in parallel by the crossbar array, e.g., 100. If, for example, a single crossbar array is used, all elements of the output matrix may be obtained by processing (e.g., sequentially) each of the subsets of dot product operations in the crossbar array. For example, to perform two subsets of dot product operations, all the convolution matrices of the above mentioned subsets are stored in a crossbar array, and the two input vectors of the above mentioned subsets are input into the crossbar array consecutively.

[0045] Each dot product operation in the set of dot product operations includes an input sub-matrix of the input matrix 321 and at least one other convolution matrix. Each dot product produces one element of the output matrix 323. The input sub-matrix is

[0046]

number

[0047] where subx in <x in , and suby in <y inA dot product operation is a process of locally multiplying like entries of two matrices and summing the results of the addition. Each dot product operation of a set of dot product operations may include an input sub-matrix having a size identical to the size of the convolution matrix. The input sub-matrices for different dot products may share elements. The terms "input sub-matrix" and "convolution matrix" are used for naming purposes to distinguish the first (left operand) and second (right operand) operands of a dot product operation.

[0048] 3 shows input sub-matrices 301 and 302 and two corresponding convolution matrices 303 and 307. As shown in FIG. 3, the two input sub-matrices 301 and 303 are an input matrix 321 with depth din=1 that can be used to obtain elements of an output matrix 323 with depth dout=1. The two input sub-matrices 301 and 302 are respectively

[0049]

number

[0050] where subx in <x in , and suby in <y in3 shows two dot product operations. In this embodiment of FIG. 3, the first dot product operation includes input sub-matrix 301 and convolution matrix 305. The second dot product operation includes input sub-matrix 303 and convolution matrix 307. For illustrative purposes, convolution matrices 305 and 307 are identical, but they may be different. Input sub-matrices 301 and 303 have elements a2, a3, a5, a6, a8, and a9. This is also illustrated on input matrix 321. Thus, input sub-matrices 301 and 303 have the following different elements: a1, a2, a3, a4, a5, a6, a7, a8, a9, b1, b2, and b3.

[0051] In one embodiment, the first dot product operation and the second dot product operation can be part of the (total) convolution of the input matrix 321 of each kernel 305 and 307. For example, the convolution of the kernel 305 with the input matrix 321 can include the first dot product operation and an additional dot product operation obtained by sliding the kernel 305 over the input matrix 321. This is particularly advantageous since the method can be used for convolutions included in the operation of a neural network. Thus, according to the embodiment of FIG. 3, the set of dot product operations includes a subset of two dot product operations, the first and the second dot product operations. These two dot product operations can be performed in parallel to calculate the two elements of the output matrix 323, and thus speed up the computation process compared to a method in which each element of the matrix is ​​calculated separately.

[0052] Referring back to FIG. 2, the method may optimally use a crossbar array, such as crossbar array 100 described with reference to FIG. 1, to perform a set of dot product operations. The set of dot product operations may be performed by computing multiple subsets of dot product operations, such as the subsets of dot product operations defined in FIG. 3, with first and second dot product operations. For example, in FIG. 3, two subsets of dot product operations are performed simultaneously using crossbar array 300. In accordance with the embodiment of FIG. 3, the method may use a crossbar array to compute the following two results: a1*k1+a2*k2+a3*k3+a4*k4+a5*k5+a6*k6+a7*k7+a8*k8+a9*k9 is the result of the first dot product operation, and a2*k1+a3*k2+b1*k3+a5*k4+a6*k5+b2*k6+a8*k7+a9*k8+b3*k9 is the result of the second dot product operation.

[0053] To compute a subset of dot product operations, an input vector is provided that includes different elements of the input sub-matrix. The different elements are arranged in the input vector according to a predefined order, so that the elements of the input vector can be configured to be input simultaneously to the crossbar array into a corresponding sequence of row lines of the crossbar array. For example, if the input vector has five elements, the five elements can be input at one time to each of five consecutive row lines of the crossbar array. The five consecutive row lines can be the first five row lines 102.1-5, or another sequence of five consecutive row lines of the crossbar array. For example, the first element of the input vector can be input to a given row line, e.g., the first row line 102.1 of the crossbar array, the second element of the input vector can be input to the subsequent row line, e.g., the second row line 102.2 of the crossbar array, and so on. According to the embodiment of FIG. 3, the input vector 310 may include different elements of the input sub-matrices 301 and 303: a1, a2, a3, a4, a5, a6, a7, a8, a9, b1, b2, and b3.

[0054] Depending on the position and order of the different elements in the input vector 310, the convolution matrix may be stored in the crossbar array accordingly in step 201. For example, this may be done by rearranging the different elements in the input vector multiple times to obtain a multiply rearranged input vector. For each of the multiply rearranged input vectors, a corresponding set of storage locations for the convolution matrix in the crossbar array is determined. This gives a multiplicity of sets of storage locations. For example, for a given rearranged input vector, storing the convolution matrix in a corresponding set of storage locations allows for computing a set of dot product operations by inputting the given rearranged input vector into respective row lines of the crossbar array. Each of the sets of storage locations may occupy a surface of the crossbar array. In step 201, the convolution matrix may be stored in a set of storage locations that occupies a minimum surface.

[0055] An input vector with different elements can be input to the crossbar array in step 203, and a subset of the dot product operations can be performed using the stored convolution matrix. For example, each element of the input vector can be input to a corresponding row line of the crossbar array. The column outputs of the crossbar array allow for the results of the subset of the dot product operations to be obtained.

[0056] According to the embodiment of Fig. 3, the convolution matrices 305 and 307 may be stored in two consecutive row lines of the crossbar array 300, and the input vector includes different elements in the following order: b1, b2, b3, a2, a5, a8, a3, a6, a9, a1, a4 and a7. The output px1 of the first column is the first result, a1*k1+a2*k2+a3*k3+a4*k4+a5*k5+a6*k6+a7*k7+a8*k8+a9*k9, and the output px2 of the second column is the second result, a2*k1+a3*k2+b1*k3+a5*k4+a6*k5+b2*k6+a8*k7+a9*k8+b3*k9.

[0057] FIG. 4 is a flow chart of a method for performing at least a part of the estimation process of a convolutional neural network (CNN). For the purpose of simplicity, the method of FIG. 4 is described with reference to the example of FIGS. 5A-B, but without being limited thereto. The CNN receives as input, for example, an input feature map 501 of depth din. The input feature map 501 may include channels or layers of din, for example the feature map may include color channels with din=3. The input feature map 501 may therefore be referred to as a multi-dimensional matrix. The estimation process of the CNN includes a convolution of the input feature map 501 with a kernel of size k*k, which gives an output feature map 503 having depth dout. The number of kernels may for example be equal to dout. The output feature map 503 may include channels of dout. The output feature map 503 is therefore also a multi-dimensional matrix. For ease of explanation, the output feature map 503 is shown as including 8×8 pixels, where a pixel of the plurality of pixels comprises the dout element of the output feature map 503. Figure 5A shows two pixels pix1 and pix2. The first pixel pix1 has dout values ​​(or elements) pix1_1, pix1_2...pix1_dout in each channel of the output feature map 503. The second pixel pix2 has dout values ​​(or elements) pix2_1, pix2_2...pix2_dout in each channel of the output feature map 503.

[0058] 5B shows four pixels pix1, pix2, pix3, and pix4. The first pixel pix1 has dout values ​​(or elements) pix1_1, pix1_2...pix1_dout in each channel of the output feature map 503. The second pixel pix2 has dout values ​​(or elements) pix2_1, pix2_2...pix2_dout in each channel of the output feature map 503. The third pixel pix3 has dout values ​​(or elements) pix3_1, pix3_2...pix3_dout in each channel of the output feature map 503. The fourth pixel pix4 has dout values ​​(or elements) pix4_1, pix4_2...pix4_dout in each channel of the output feature map 503.

[0059] For example, to obtain pixel values ​​(e.g., pix1_1 and pix2_1) of a single channel of the output feature map 503, the following is performed: A k×k kernel can be shifted through the channels of the input feature map 501 to perform the convolution. This is done for each pixel and each channel in a dot product operation between one kernel and one sub-matrix of size k*k*din. Following the example of FIGS. 5A-B, the input feature map 501 has 10×10 pixels in each channel, and shifting a 3×3 kernel over the channels gives 64 dot product operations for each channel of the output feature map (dot product operations of 3×3 pixels with a 3×3 kernel). Each dot product operation of the input feature map 501 can include, for example, an input sub-matrix 505 with a size of 3×3×din (or 3×3 pixels). For example, to obtain pixel value pix1_1 of the first channel of the output feature map 503, one dot product operation can be performed on each input sub-matrix 505. For example, the dot product operation can be performed using the same or different 3×3 kernels for each channel of the sub-matrix 505. 64 dot product operations are performed to obtain a single channel of the output feature map 503. Thus, dout*64 dot product operations are the set of dot product operations involved in the matrix convolution on the input feature map 501 to obtain the output feature map 503.

[0060] FIG. 5A illustrates one mapping method where a dot product operation of dout*2 can be computed by a crossbar array in one time step (e.g., one clock cycle). FIG. 5B illustrates one mapping of a convolution matrix onto a crossbar array where a dot product operation of dout*4 can be computed in one time step. The dot product operation of dout*2 shown in FIG. 5A can include a dout*din*2 kernel to obtain two pixels pix1 and pix2 of the output matrix. The dot product operation of dout*4 in FIG. 5B can include a dout*4 kernel to compute pixels pix1, pix2, pix3, and pix4. Thus, the difference between FIG. 5A and FIG. 5B is the different subset of dot product operations to be performed on a single crossbar array. In FIG. 5A, two pixels, pix1 and pix2, are calculated by crossbar array 520, whereas in FIG. 5B, four pixels, pix1, pix2, pix3 and pix4, are calculated by crossbar array 620.

[0061] In step 401, it may be determined which subsets of the total set of dot product operations should be performed together or in parallel using a single crossbar array. For example, in FIG. 5A, a set of dout*2 dot product operations is determined or selected that includes input matrices 505 and 507. In FIG. 5B, a set of dout*4 dot product operations is determined or selected that includes input matrices 505, 507, 509, and 511.

[0062] In step 403, distinct elements of the input sub-matrix included in the determined subset of dot-product operations can be identified. In the example of FIG. 5A, the number of distinct elements in the input sub-matrix of the dot-product operation of dout*2 can be equal to din*k*+k*din, but in general, the number of distinct elements in the input sub-matrix of the input feature map 501 can be equal to k*+d*din, as shown in FIGS. 5A-B.

[0063]

number

[0064] where N is the number of pixels to be calculated, for example in FIG. 5A N=2 and in FIG. 5B N=4.

[0065] In step 405, all kernels for execution of the determined subset of dot product operations are stored in a crossbar array.

[0066] For example, FIG. 5A shows a crossbar array 520 with the number of row lines, din*k*k+k*din, corresponding to the identified different elements, and the number of columns corresponding to the channels of pixels to be calculated. For example, in FIG. 5A, the number of column lines is 2*dout. Each crossbar array stores k*k*din kernel elements (e.g., for din=3, a kernel can be stored in each row). The first pixel pix1 for the dout channel can be obtained by column 521 (e.g., the first column of columns 521 can provide the value pix1_1, the second column of columns 521 can provide the value pix1_2, etc.), and the value pix2 of the second pixel for all dout channels can be obtained by column 522 (e.g., the first column of columns 522 can provide the value pix2_1, the second column of columns 522 can provide the value pix2_2, etc.). The area occupied by the kernel in Figure 5A is defined by rectangles 531 and 532, and the remaining elements of the crossbar array can be set to zero as shown in Figure 5A. The area of ​​the crossbar array shown in Figures 5A-B is for illustration purposes only. For example, the size of rectangles 531, 532 and 631-634 is determined by the size of the kernel k, and the values ​​of din and dout.

[0067] For example, FIG. 5B shows a crossbar array 620 with a number of row lines corresponding to the number of identified distinct elements, din*k*k+3*k*din, and a number of columns corresponding to the number of channels of pixels to be calculated. For example, in FIG. 5B, the number of column lines is 4*dout. As shown in FIG. 5B, the value of the first pixel pix1 can be obtained by column 621, the value of the second pixel pix2 can be obtained by column 622, the value of the third pixel pix3 can be obtained by column 623, and the value of the fourth pixel pix4 can be obtained by column 624. The area occupied by the kernel of the crossbar array of FIG. 5B is defined by rectangles 631, 632, 633, and 634, and the remaining elements of the crossbar array can be set to zero as shown in FIG. 5B.

[0068] Thus, as shown in Figures 5A and 5B (and also Figure 3), the kernels are stored on the crossbar array in a surface-efficient manner so that they occupy an optimal surface area of ​​the crossbar array while still allowing the performance of a set of dot product operations.

[0069] In step 407, the input vectors of different elements can be input to the crossbar array 520 to collect the computation results of the determined subset of dot product operations from the crossbar array. The input vectors can be input to the crossbar array simultaneously so that the crossbar array can perform all the subset of dot product operations in, for example, one clock cycle.

[0070] In the example of Figure 5A, each row may output pixel values ​​for a single channel of output feature map 503, e.g., a value pix1_1 may be output for the first column of columns 521 of crossbar array 520. In the example of Figure 5B, each column may output pixel values ​​for a single channel of output feature map 503, e.g., a value pix2_1 may be output for the first column of columns 622 of crossbar array 620. The pixel values ​​output by the crossbar array are read to provide elements of an output matrix.

[0071] The method of Figure 4 can be iterated for further subsets of the set of dot product operations. For example, if the subset of dot product operations determined in the first implementation includes a dot product operation of dout * 2, the method can be iterated for another dot product operation of dout * 2, for example, covering input matrices 509 and 511 of Figure 5B. In a given iteration of the method, kernel values ​​stored in the crossbar array can be deleted (or overwritten) so that new values ​​can be stored in the crossbar array.

[0072] FIG. 6, for example, illustrates a method for selecting a subset of the dot product operation of FIG. 2. Like FIGS. 5A and 5B, FIG. 6 shows an input feature map 601 and an output feature map 603. Output feature map 603 includes one horizontal and vertical sequence of pixels that can be processed in accordance with the present subject matter. The number of pixels to be processed by a single crossbar array can be determined by fixing the vertical direction to a fixed number of pixels and selecting the number of pixels along the other directions so that they can be performed in parallel using a single crossbar array.

[0073] For example, the size d1 (in the vertical direction) is fixed to two pixels pix1 and pix5, and other pixels in the vertical direction are chosen or selected. For example, if it is decided to calculate four pixels, the calculations of pix2 and pix6 (following horizontally) are added to the calculation of pix1 and pix5. For example, if it is decided to calculate eight pixels (shown in FIG. 6), pix2, pix3, pix7, pix4 and pix8 (following horizontally) are added to pix1 and pix5 to calculate their values.

[0074] The total number of pixels to be calculated determines the subset of dot product operations to be performed by the crossbar array. For example, by finding d1 (i.e., one direction fixed), pixels along other directions can be calculated in parallel. In the embodiment of FIG. 6, dout*8 dot product operations can be performed. dout*8 dot product operations are stored in the crossbar array 720. As shown in FIG. 6, some kernels do not share elements of the input vector, but the total area occupied by multiple kernels on the crossbar area can still be the optimum area. Like FIGS. 5A-B, FIG. 6 shows the area occupied by the convolution matrix as a rectangle with a different display format, and the remaining elements of the crossbar array not covered by the area are set to zero.

[0075] FIG. 7 shows a graphical representation of the ResNet 700 architecture. FIG. 7 shows that TesNet has five different levels 701-705. Each of the levels 701-705 can contain a multidimensional matrix of different sizes. For example, as shown in FIG. 7, the output matrix 701 of level 1 and 702 of level 2 have 16 channels and contain kernels of size 3×3. The crossbar array associated with each layer 710 outputs a number of pixels of the output matrix. For example, the crossbar array of layer 710 of level 2 can output at least two pixels, each pixel having 16 values ​​or elements of the output matrix. This means that if the interconnect, shown as a line joining two consecutive layers 710, delivers data in one time step, its bandwidth requirement will be a multiple of 16. FIG. 7 shows that the maximum bandwidth is used for level 4, i.e., a multiple of 64.

[0076] The interconnects between the crossbar arrays in layer 710 of the ResNet can be designed based on the maximum bandwidth, but some interconnects may be lower than that of the maximum bandwidth. As a result, the crossbar array in level 2, 702, can be used to compute as many as 4 (=64 / 16) pixels, and the crossbar array in level 3, 703, can compute as many as 2 (=64 / 32) pixels. This approach allows the maximum bandwidth of 64 to be used at all times.

[0077] In one embodiment, a CNN can be provided in which each layer is accompanied by a crossbar array. The training or estimation of the CNN can include layer operations, for example, to generate an output feature map. The crossbar array of the CNN can be configured to compute each pixel of the output feature map using the method, so that the pixels can be generated in parallel by each crossbar array, and in number of pixels, the bandwidth is constant throughout the entire CNN network.

[0078] Features of the invention are described herein with reference to flowchart illustrations or block diagrams of methods or devices or combinations thereof and systems according to embodiments of the invention, it will be understood that the flowchart illustrations or block diagrams or combinations thereof and the blocks or block diagrams within the flowchart illustrations or combinations of both may be implemented by computer readable program instructions.

[0079] Referring to FIG. 8, the system 1000 includes a computer system or computer 1010 shown in the form of a general computing device. The methods described herein may be implemented in, for example, a program(s) 1060 (FIG. 8) embodied on a computer-readable storage device, for example generally referred to as a memory 1030, and more specifically, a computer-readable storage medium 1050 shown in FIG. 8. For example, the memory 1030 may include a storage medium 1034, such as a RAM (random access memory) or a ROM (read only memory) and a cache memory 1038. The program 1060 may be executed (to execute program steps, codes or program codes) by a processing unit or processor 1020 of the computer system 1010. Additional data storage may be embodied as a database 1110, which may also include data 1114. The computer system 1010 and programs 1060 shown in FIG. 8 are generally representative of computers and programs that may be provided to users locally or as a remote service (e.g., cloud-based services), and in further embodiments may be provided using a website-accessible communications network 1200 (e.g., internet interacting network, or cloud services). Computer system 1010 is also generally understood herein to include one or more servers, computing devices such as laptop or desktop computers, alone or as part of a data center, or computers contained within such devices. The computer system may include a network adapter / interface 1026 and input / output (I / O) interface(s) 1022. The I / O interface 1022 allows for input and output of data to and from external devices 1074 that may be connected to the computer system.The network adapter / interface 1026 can provide communication between the computer system and a network, generally depicted as a communications network 1200 .

[0080] The computer 1010 may be described in the general context of computer system executable instructions, such as program modules executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Method steps and system components and techniques may be embodied in program modules for performing the tasks of the respective steps of the methods and systems. Modules are generally represented in FIG. 8 as program modules 1064. Programs 1060 and program modules 1064 may execute particular program steps, routines, subroutines, instructions, or code.

[0081] The disclosed method may operate locally on a device, such as a mobile device, or remotely, and may be accessed using a communications network 1200, for example on a server 1100. The programs or executable instructions may also be provided as a service by a provider. The computer 1010 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through the communications network 1200. In a distributed cloud computing environment, program modules may be located in storage media, including memory storage devices, in both local and remote computer systems.

[0082] 8, system 1000 includes a computer system 1010 shown in the form of a general purpose computing device along with exemplary peripheral devices. Components of computer system 1010 include, but are not limited to, one or more processors or processing units 1020, a system memory 1030, and a bus 1014 that couples various system components including the system memory 1030 to the processor 1020.

[0083] Bus 1014 may represent any of several types of one or more bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro-Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Interconnect (PCI) bus.

[0084] The computer 1010 typically includes a variety of computer readable media. Such media may be any available media accessible by the computer 1010 (e.g., a computer system or server) and includes volatile and non-volatile media, removable and non-removable media. The computer memory 1030 may include additional computer readable media 1034, such as volatile memory, such as random access memory (RAM), or cache memory 1038, or both. The computer 1010 may further include other removable / non-removable, volatile / non-volatile computer storage media, which in one embodiment is a portable computer readable recording medium 1072. In one embodiment, the computer readable recording medium 1050 may provide for reading from and writing to a non-removable, non-volatile magnetic medium. The computer readable recording medium 1050 may be embodied as, for example, a hard drive. Additional memory and data storage may be provided, such as a storage system 1110 (e.g., a database) for storing data 1114 and communicating with the processing unit 1020. The database may be stored on or part of the server 1100. Although not shown, an optical disk drive may be provided that can read from and write to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such an example, each may be connected to the bus 1014 by one or more media interfaces. As further shown and described below, the memory 1030 may include at least one program product that includes one or more program modules configured to perform functions of embodiments of the present invention.

[0085] The methods of the present disclosure may be embodied in, for example, one or more computer programs, generally referred to as program(s) 1060, stored in memory 1030 in computer readable storage medium 1050. Program modules 1064 may generally perform the functionality and / or methodology of embodiments of the present invention as described herein. One or more programs 1060 may be stored in memory 1030 and executed by processing unit 1020. By way of example, memory 1030 may store, on computer readable storage medium 1050, an operating system 1052, one or more application programs 1050, other program modules, and program data. The programs 1060, operating system 1052, and application programs 1054 stored on computer readable storage medium 1050 may be similarly executed by processing unit 1020.

[0086] The computer 1010 may also communicate with one or more external devices 1074, such as a keyboard, pointing device, display 1080, which may allow a user to interact with the computer 1010 or any device (e.g., network card, modem, etc.) that allows the computer 1010 to communicate with one or more computing devices, or both. Such communication may occur through an input / output (I / O) interface 1022. Additionally, the computer 1010 may communicate with one or more networks 1200, such as a local area network (LAN), a general wide area network (WAN), or a public network (Internet), or combinations thereof, through a network adapter / interface 1026. As shown, the network adapter 1026 communicates with other components of the computer 1010 through a bus 1014. It should be understood that other hardware and / or software components, not shown, could be used in combination with the computer 1010. Examples include, but are not limited to, microcode, device drivers 1024, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0087] A computer or program running on computer 1010 can communicate with a server embodied as server 1100 via one or more communication networks embodied as communication network 1200. Communication network 1200 can include network links including, for example, communication media and wireless, wired, or fiber optics, and routers, firewalls, switches, and gateway computers. A communication network can include connections such as wired, wireless communication links, or fiber optic cables. A communication network can represent a worldwide collection of networks and gateways such as the Internet network that communicate with each other using various protocols such as Lightweight Directory Access Protocol (LDAP), Transport Control / Internet Protocol (TCP / IP), Hypertext Transport Protocol (HTTP), Wireless Application Protocol (WAP), and the like. A network can also include many networks of different types, such as the Internet, a local area network (LAN), or a wide area network (WAN).

[0088] In one embodiment, the computer may use a network that allows access to websites on the World Wide Web using the Internet. In one embodiment, the computer 1010 may include a mobile device and may use a communication system or network 1200 that may include the Internet or a public switched telephone network, such as a cellular network. The PSTN may include telephone lines, fiber optic cables, microwave communication links, cellular networks, and satellite communications. The Internet may facilitate many search and text techniques, such as using text messages (SMS), multimedia messaging service (MMS) (related to SMS), email, or a cellular phone or laptop computer to submit a query to a search engine via a web browser. The search engine may allow retrieval of search results, i.e., links to websites, documents, or other downloadable data corresponding to the query, and may also provide the search results to the user via the device, for example, as a web page.

[0089] Although this disclosure includes details about cloud computing, it should be understood that implementation of the teachings referenced herein is not limited to a cloud computing environment, rather, the disclosed environment can be implemented in combination with any other type of computing environment now known or developed in the future.

[0090] Cloud computing is a service delivery model for on-demand network access with the convenience of accessing a shared pool of rapidly provisioned and open configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) with minimal administrative effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0091] Its features are as follows:

[0092] On-Demand Self-Service: Cloud consumers are automatically provisioned with computing capacity - server time and network storage - on an as-needed basis without the need for human interaction with the service provider.

[0093] Pervasive Network Access: Capabilities are available over the network and accessed through standard mechanisms facilitating use by different thin- and thick-client platforms (eg, mobile phones, laptops, and PDAs).

[0094] Resource Sharing: Using a multi-tenant model, a provider's computing resources are shared to serve multiple consumers, with different physical and virtualized resources dynamically allocated and reallocated as needed. A sense of location independence exists, such that consumers generally have no control or knowledge of the exact location (e.g., country, state, or data center) of the resources provided, but can specify the location at a higher layer of abstraction.

[0095] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, to quickly scale out and quickly release to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time.

[0096] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services being used.

[0097] The service model is as follows:

[0098] Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for limited user-specific application configuration settings.

[0099] Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer has created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, potentially, the configuration of the application hosting environment.

[0100] Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provision of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0101] The layout model is as follows:

[0102] Private Cloud: The cloud infrastructure operates solely for one organization. It can be managed by that organization or a third party and can exist on-premises or off-premises.

[0103] Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by those organizations or by a third party and can exist on-premises or off-premises.

[0104] Public Cloud: Cloud infrastructure is made available to the public or large industry groups and is owned by organizations that sell cloud services.

[0105] Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound to each other by standardized or proprietary technologies that allow portability of data and applications (e.g., cloud bursting for load balancing between clouds).

[0106] Cloud computing environments are service-oriented with a focus on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, which contains multiple interconnected nodes.

[0107] FIG. 9 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, communicate. The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof (not shown). This allows the cloud computing environment 50 to provide an infrastructure, platform, or software-as-a-service for cloud consumers to eliminate the need to maintain resources on local computing devices. It will be understood that the types of computing devices 54A-N shown in FIG. 9 are for illustrative purposes only, and that the computing node 10 and cloud computing environment 50 can communicate with any type of computerized device through any type of network or addressable network connection (e.g., using a web browser), or both.

[0108] 10, a set of functional abstraction layers provided by cloud computing environment 50 (FIG. 9) is shown. It should be understood that the components, layers, and functions shown in FIG. 10 are intended to be illustrative only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0109] The hardware and software layer 60 includes hardware and software components. Examples of the hardware components may include a mainframe 61, a plurality of servers based on a RISC (reduced instruction set computer) architecture 62, a plurality of servers 63, a plurality of blade servers 64, a plurality of storage devices 65, and a network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0110] The visualization layer 70 provides an abstraction layer from which embodiments of virtual entities, described below; virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0111] In one embodiment, the management layer 80 may provide the following functions: A resource provisioning unit 81 provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. A metering and pricing unit 82 provides cost tracking as resources are used within the cloud computing environment and provides billing or charging for the consumption of these resources. In one embodiment, these resources may include application software licenses. A security unit provides identification and authentication of cloud consumers and tasks as well as protection of data and other resources. A user portal unit 83 provides accessibility to the cloud computing environment and system administrators for consumers. A service level management unit 84 provides allocation and management of cloud computing resources to meet required service levels. A service level agreement (SLA) planning and fulfillment unit 85 provides pre-provisioning and acquisition of cloud computing resources as future requests are required according to SLAs.

[0112] The Workload Layer 90 provides examples of functionality for utilizing a cloud computing environment. Examples of workloads and functionality provided by this layer may include mapping and navigation 91; software development and lifetime management 92; virtual classroom instructional delivery 93; data analytics processing 94; transaction processing 95; and data classification 96.

[0113] The programs described herein are identified based on the application and implemented for that application in a particular embodiment of the invention, however, it should be understood that the names of any particular programs herein are merely for convenience, and thus the invention should not be limited to the use of any particular application alone, either identified or implied by such names, or both.

[0114] The present invention may be a system, method, or computer program product, or combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable recording medium (or media) having computer-readable program instructions thereon for causing a processor to perform features of the present invention.

[0115] A computer readable recording medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electro-magnetic recording device, a semiconductor recording device, or any suitable combination thereof. More specific examples of computer readable recording media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0116] The computer program instructions described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper communication cables, fiber optic communication, wireless communication routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0117] The computer readable program instructions for carrying out the operations of the present invention can be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more procedural programming languages, such as Smalltalk®, object oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions can be executed entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection can be to an external computer (e.g., through an Internet service provider). In some embodiments, computer readable program instructions may be executed by electrical circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and state information of the computer readable program instructions may be used to personalize the electrical circuitry to perform features of the present invention.

[0118] The embodiments of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer readable recording media and computer programs according to embodiments of the invention. It is understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer readable program instructions.

[0119] These computer readable program instructions can be provided to a computer processor or other programmable data processing device to create a machine, and the instructions, when executed by the computer processor or other programmable data processing device, generate means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof. These computer readable program instructions can also be stored on a computer readable recording medium that directs a computer, programmable data processing device, and other devices, or a combination thereof, to function in a particular manner, and a computer readable recording medium having instructions stored therein constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof.

[0120] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams or in a combination thereof.

[0121] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer programs according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may be performed in a manner that is actually performed as one step, may be performed simultaneously, substantially simultaneously, partially or completely overlapped in time, or the blocks may be performed in reverse order, depending on the functionality involved. Also, the block diagrams and / or flowchart illustrations and / or combinations of the blocks in the block diagrams and flowchart illustrations may be implemented by a special purpose hardware-based system that performs a particular function or operation or performs a combination of specific purpose hardware and computer instructions.

Claims

1. 1. A method for performing matrix convolution on a multi-dimensional input matrix to obtain a multi-dimensional output matrix, the matrix convolution comprising a set of parallel dot product operations to obtain all elements of the output matrix, each dot product operation of the set of parallel dot product operations comprising an input sub-matrix of the input matrix and at least one convolution matrix, the method comprising: providing a memristor-based crossbar array configured to perform vector-matrix multiplication; computing the subset of the set of parallel dot product operations by storing the convolution matrices of the subset of the set of parallel dot product operations in the crossbar array; and inputting, into said crossbar array, an input vector that includes all distinct elements of said input sub-matrices of said subset of said set of parallel dot product operations; A method comprising:

2. Computing the subset of the set of parallel dot product operations further comprises: selecting the subset of the set of parallel dot product operations, wherein the computation of the subset of the set of parallel dot product operations provides elements along two dimensions of the output matrix, and each selected subset of the set of parallel dot product operations includes a different input vector. The method of claim 1.

3. Computing the subset of the set of parallel dot product operations further comprises: selecting the subset of the set of parallel dot product operations, wherein the computation of the subset of the set of parallel dot product operations provides elements along three dimensions of the output matrix, and each selected subset of the set of parallel dot product operations includes a different input vector. The method of claim 1.

4. training or inference of a convolutional neural network includes layer operations that can be computed by the memristor-based crossbar array at each layer of the convolutional neural network, and the matrix convolution is a layer operation for a given layer of the convolutional neural network; The method of claim 1.

5. Providing the memristor-based crossbar array configured to perform the vector-matrix multiplication further comprises: providing a further memristor-based crossbar array and associating each further layer of the convolutional neural network with the memristor-based crossbar array; interconnecting the memristor-based crossbar array for execution in a pipelined manner; and performing said calculations for each further layer of said convolutional neural network using a respective subset of said set of parallel dot product operations.

5. The method of claim 4, comprising:

6. The subset of the set of parallel dot-product operations computed by each memristor-based crossbar array has the same bandwidth requirement for each interconnect between the interconnected memristor-based crossbar arrays. The method according to claim 5.

7. The memristor crossbar array includes row lines, column lines crossing the row lines, and a plurality of resistive memory elements coupled between the row lines and the column lines at intersections formed by the row and column lines, the resistive memory elements of the plurality of resistive memory elements representing values ​​of elements of a matrix. The method of claim 1.

8. Storing the convolution matrix includes: for each dot product operation of the subset of the set of parallel dot product operations, storing all elements of a convolution matrix included in the dot product operation in a plurality of resistive memory elements of a respective single column line of the crossbar array. The method of claim 7.

9. Storing the convolution matrix includes: storing all elements of the convolution matrix involved in each dot product operation of said subset in each row line; 8. The method of claim 7, comprising:

10. Storing the convolution matrix includes: identifying a group of convolution matrices from the plurality of convolution matrices to be multiplied by the same input sub-matrix, storing all elements of each convolution matrix of the group within a column line of the crossbar array; and Repeating said identifying and said storing for zero or more additional groups of convolution matrices. The method of claim 1 , comprising:

11. The input and output matrices include values ​​of pixels of an image or activations from layers of a convolutional neural network, and the convolutional matrices are kernels. The method of claim 1.

12. A memristor-based crossbar array for performing matrix convolution on a multidimensional input matrix to obtain a multidimensional output matrix, the matrix convolution including a set of parallel dot product operations to obtain all elements of the output matrix, each dot product operation of the set of parallel dot product operations including an input sub-matrix of the input matrix and at least one convolution matrix, the crossbar array configured to store the convolution matrices within the crossbar array, and an input vector including all distinct elements of the input sub-matrix is ​​input to the crossbar array to perform a subset of the dot product operations of the set of parallel dot product operations. Memristor-based crossbar array.

13. 1. A computer program for performing a matrix convolution on a multi-dimensional input matrix to obtain a multi-dimensional output matrix, the matrix convolution comprising a set of the sets of parallel dot product operations to obtain all elements of the output matrix, each dot product operation of the set of the sets of parallel dot product operations comprising an input sub-matrix of the input matrix and at least one convolution matrix, the computer program comprising: providing a memristor-based crossbar array configured to perform vector-matrix multiplication; computing the subset of the set of sets of parallel dot product operations by storing the convolution matrices of the subset of the set of parallel dot product operations in the crossbar array; and inputting, into said crossbar array, an input vector that includes all distinct elements of said input sub-matrices of said subset of dot product operations; A computer program that executes the following:

14. Computing the subset of the set of parallel dot product operations further comprises: selecting the subset of the set of parallel dot product operations, wherein the computation of the subset of the set of parallel dot product operations provides elements along two dimensions of the output matrix, and each selected subset of the set of parallel dot product operations includes a different input vector.

14. A computer program product according to claim 13.

15. Computing the subset of the set of parallel dot product operations further comprises: selecting the subset of the set of parallel dot product operations, wherein the computation of the subset of the set of parallel dot product operations provides elements along three dimensions of the output matrix, and each selected subset of the set of parallel dot product operations includes a different input vector.

14. A computer program product according to claim 13.

16. training or inference of a convolutional neural network includes layer operations that can be computed by the memristor-based crossbar array at each layer of the convolutional neural network, and the matrix convolution is a layer operation for a given layer of the convolutional neural network; 14. A computer program product according to claim 13.

17. Providing the memristor crossbar array configured to perform the vector-matrix multiplication further comprises: providing a further memristor-based crossbar array and associating each further layer of the convolutional neural network with the memristor-based crossbar array; interconnecting the memristor-based crossbar array for execution in a pipelined manner; and performing said calculations for each further layer of said convolutional neural network using a respective subset of said set of parallel dot product operations.

17. The computer program product of claim 16, comprising:

18. The subset of the set of parallel dot-product operations computed by each memristor-based crossbar array has the same bandwidth requirement for each interconnect between the interconnected memristor-based crossbar arrays.

18. A computer program product according to claim 17.

19. The memristor crossbar array includes row lines, column lines crossing the row lines, and a plurality of resistive memory elements coupled between the row lines and the column lines at intersections formed by the row and column lines, the resistive memory elements of the plurality of resistive memory elements representing values ​​of elements of a matrix.

14. A computer program product according to claim 13.

20. Storing the convolution matrix includes: for each dot product operation of the subset of the set of parallel dot product operations, storing all elements of a convolution matrix included in the dot product operation in a plurality of resistive memory elements of a respective single column line of the crossbar array.

14. A computer program product according to claim 13.

Images