Crossbar circuit for performing convolution operations

Abstract

An apparatus for performing a convolution operation is provided according to some embodiments of the present disclosure. The apparatus includes a first crossbar circuit including a first plurality of crosspoint elements, a second crossbar circuit including a second plurality of crosspoint elements, and word line logic that applies input signals to the first crossbar circuit and the second crossbar circuit. The word line logic is configured to provide input signals representing input data to be convolved using one or more two-dimensional convolution kernels and one or more depthwise convolution kernels. The first crossbar circuit is configured to output a first plurality of output signals representing the convolution of the input data with the two-dimensional convolution kernels. The second crossbar circuit is configured to output a second plurality of output signals representing the convolution of the input data with the depthwise convolution kernels.

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims priority based on U.S. Patent Application No. 17 / 504,478, filed on October 18, 2022, under the title "Crossbar Circuits For Performing Convolution Operations", which is incorporated herein by reference in its entirety.

[0002] Implementations of the present disclosure generally relate to crossbar circuits, and more specifically, to crossbar circuits for performing regular 2D convolution operations and depth - wise convolution operations.

Background Art

[0003] A crossbar circuit can refer to a circuit structure involving the interconnection of conductive lines sandwiching memory elements such as resistive switching materials at the intersections. The resistive switching material can include, for example, memristors (also sometimes called resistive random - access memory (RRAM or ReRAM)). Crossbar circuits can be used in implementations such as in - memory computing applications, non - volatile solid - state memories, image - processing applications, neural networks, etc.

Summary of the Invention

Means for Solving the Problems

[0004] The following is a simplified summary of the present disclosure to provide a basic understanding of some aspects of the present disclosure. This summary is not an overview of the broad scope of the present disclosure. It is not intended to identify key or important elements of the present disclosure, nor to delineate the scope of a particular implementation of the present disclosure or the scope of the claims. It is simply intended to present some concepts of the present disclosure in a simplified form as a prelude to the more detailed description provided later.

[0005] According to one or more aspects of the present disclosure, an apparatus for performing a depth convolution operation is provided. The apparatus includes a plurality of crosspoint elements connecting a plurality of word lines and a plurality of bit lines; and a plurality of selection lines. The plurality of selection lines include a first selection line connecting a first group of the plurality of crosspoint elements and a second selection line connecting a second group of the plurality of crosspoint elements. The first group of the plurality of crosspoint elements includes a first crosspoint element connecting a first bit line of the plurality of bit lines and a first word line of the plurality of word lines, and a second crosspoint element connecting a second bit line of the plurality of bit lines and a second word line of the plurality of word lines. The second group of the plurality of crosspoint elements includes a third crosspoint element connecting a second word line and a first bit line, and a fourth crosspoint element connecting a third word line and a second bit line.

[0006] In some embodiments, the device further includes selection logic configured to selectively enable one or more of the multiple crosspoint elements via the multiple selection lines.

[0007] In some embodiments, the selection logic is further configured to enable the first set of crosspoint elements via the first selection line.

[0008] In some embodiments, the selection logic is further configured to enable the second set of crosspoint elements via the second selection line.

[0009] In some embodiments, the apparatus further includes programming logic configured to map a first plurality of elements of the depth convolution kernel to a first plurality of crosspoint elements, and to map a second plurality of elements of the depth convolution kernel to a second plurality of crosspoint elements.

[0010] In some embodiments, the apparatus further includes word line logic connected to the plurality of word lines, the word line logic is configured to receive input data to be convolved using a plurality of depth convolution kernels, generate a plurality of input signals representing the input data, and apply the plurality of input signals to one or more crosspoint elements among the plurality of crosspoint elements which are enabled by the selection logic.

[0011] In some embodiments, the device further includes sensing logic that generates a digital output based on a plurality of output signals output via the plurality of bit lines, wherein the digital output represents the convolution of the input data and the plurality of depth convolution kernels.

[0012] In some embodiments, the first plurality of crosspoint elements further include a fifth crosspoint element connecting the third word line and the third bit line. In some embodiments, the second plurality of crosspoint elements further include a sixth crosspoint element connecting the fourth word line and the third bit line.

[0013] In some embodiments, the plurality of selection lines are not parallel to the plurality of bit lines.

[0014] In some embodiments, the plurality of selection lines are not parallel to the plurality of word lines.

[0015] In some embodiments, at least one of the plurality of crosspoint elements includes a transistor and a memristor.

[0016] In some embodiments, the first selection line and the second selection line are connected to a first selection register.

[0017] In some embodiments, the first selection line is connected to a first selection register, and the second selection line is connected to a second selection register.

[0018] According to one or more aspects of the present disclosure, a method is provided for performing depth convolution using a crossbar-based device. The method includes mapping a plurality of kernels to a plurality of crosspoint elements of a crossbar circuit; enabling the plurality of crosspoint elements via a plurality of selection lines of the crossbar circuit; applying a plurality of input signals to the plurality of crosspoint elements representing input data to be convolved using the plurality of kernels; and outputting a plurality of output signals via a plurality of bit lines of the crossbar circuit representing the convolution result of the input data and the kernels, wherein the plurality of selection lines include a first selection line connecting a first group of crosspoint elements to a second group of crosspoint elements. The first group of crosspoint elements includes a first crosspoint element connecting a first word line to a first bit line and a second crosspoint element connecting a second word line to a second bit line. The second group of crosspoint elements includes a third crosspoint element connecting the second word line and the first bit line, and a fourth crosspoint element connecting the third word line and the second bit line.

[0019] In some embodiments, the plurality of kernels includes a plurality of depth convolution kernels used to perform depth convolution operations.

[0020] In some embodiments, mapping the plurality of kernels to the plurality of crosspoint elements of the crossbar circuit includes programming the conductance value of the first crosspoint element to a first value representing the first element of the first kernel among the plurality of kernels, and programming the conductance value of the first crosspoint element to a second value representing the first element of the second kernel among the plurality of kernels.

[0021] In some embodiments, mapping the plurality of kernels to the plurality of crosspoint elements of the crossbar circuit further includes programming the conductance value of the third crosspoint element to a third value representing the second element of the first kernel, and programming the conductance value of the fourth crosspoint element to a fourth value representing the second element of the second kernel.

[0022] In some embodiments, enabling the plurality of crosspoint elements via the plurality of selection lines includes applying a first selection voltage to the first selection line to enable the first group of crosspoint elements.

[0023] In some embodiments, enabling the plurality of crosspoint elements via the plurality of selection lines includes applying a second selection voltage to the second selection line to enable the second group of crosspoint elements.

[0024] In some embodiments, applying the plurality of input signals to the plurality of crosspoint elements, which represent the input data to be convolved using the plurality of kernels, includes applying a first input signal to the first crosspoint element via a first word line of the crossbar circuit and applying a second input signal to the second crosspoint element via a second word line of the crossbar circuit, wherein the first input signal represents a first portion of the first channel of the input data and the second input signal represents a second portion of the second channel of the input data.

[0025] In some embodiments, the method further includes retrieving a first data item and a second data item stored in a contiguous address from a storage device, wherein the first input signal represents the first data item and the second input signal represents the second data item.

[0026] According to several embodiments of this disclosure, an apparatus for performing regular 2D convolution and depth convolution operations is provided. The apparatus includes a first crossbar circuit including a first plurality of crosspoint elements, a second crossbar circuit including a second plurality of crosspoint elements, and word-line logic connected to the first and second crossbar circuits via a plurality of word lines. In some embodiments, the word-line logic is configured to provide a plurality of input signals to the first and second crossbar circuits via the plurality of word lines. The input signals may represent input data to be convolved using a plurality of 2D convolution kernels and a plurality of depth convolution kernels.

[0027] Furthermore, the apparatus includes a first plurality of bit lines connected to the first crossbar circuit and a second plurality of bit lines connected to the second crossbar circuit. The first crossbar circuit is configured to output a first plurality of output signals representing the convolution result of the input data and the plurality of two-dimensional convolution kernels via the first plurality of bit lines. The second crossbar circuit is configured to output a second plurality of output signals representing the convolution result of the input data and the plurality of depthwise convolution kernels via the second plurality of bit lines.

[0028] In some embodiments, the second plurality of output signals include a first output signal output via a first bit line and a second output signal output via a second bit line. The first output signal represents the convolution of a first channel of the input data and a first depthwise convolution kernel. The second output signal represents the convolution of a second channel of the input data and a second depthwise convolution kernel.

[0029] In some embodiments, the apparatus further includes a first selection logic configured to selectively enable one or more of the first plurality of cross-point elements of the first crossbar circuit via a first plurality of selection lines, and a second selection logic configured to selectively enable one or more of the second plurality of cross-point elements of the second crossbar circuit via a second plurality of selection lines.

[0030] In some embodiments, the second plurality of selection lines include a first selection line connecting a first group of the second plurality of crosspoint elements. The first group of the second plurality of crosspoint elements includes a first crosspoint element connecting a first bit line of the plurality of bit lines to a first word line of the plurality of word lines, and a second crosspoint element connecting a second bit line of the plurality of bit lines to a second word line of the plurality of word lines.

[0031] In some embodiments, the second plurality of selection lines include a second selection line connecting a second group of the second plurality of crosspoint elements. The second group of the second plurality of crosspoint elements includes a third crosspoint element connecting a third word line to the first bit line, and a fourth crosspoint element connecting a fourth word line to the second bit line.

[0032] In some embodiments, the second selection logic includes a selection register connected to the first selection line.

[0033] In some embodiments, the selection register is further connected to one or more other selection lines from the second plurality of selection lines.

[0034] In some embodiments, the second selection logic is configured to enable the first group of the second plurality of crosspoint elements by applying a first selection voltage to the first selection line.

[0035] In some embodiments, the second selection logic is configured to enable the second group of the second plurality of crosspoint elements by applying a second selection voltage to the second selection line.

[0036] In some embodiments, the apparatus further includes a programming circuit configured to program one or more of the first plurality of crosspoint elements to store the two-dimensional convolution kernel, and to program one or more of the second plurality of crosspoint elements to store the plurality of depth convolution kernels.

[0037] In some embodiments, the device further includes sensing logic that generates a digital output based on the first plurality of output signals and the second plurality of output signals.

[0038] In some embodiments, at least one of the second plurality of crosspoint elements includes a transistor and a memristor.

[0039] In some embodiments, there is at least one other word line between the first word line and the second word line.

[0040] Methods for performing convolution are provided according to several embodiments of the present disclosure. The method includes providing a plurality of input signals to a first crossbar circuit and a second crossbar circuit of a crossbar-based device via word-line logic of the crossbar-based device, wherein the plurality of input signals represent input data to be convolved using a plurality of two-dimensional convolution kernels and a plurality of depth convolution kernels; outputting a first plurality of output signals using the first crossbar circuit representing the convolution result of the input data and the plurality of two-dimensional convolution kernels; and outputting a second plurality of output signals using the second crossbar circuit representing the convolution result of the input data and the plurality of depth convolution kernels.

[0041] In some embodiments, the method for performing the convolution further includes programming a first plurality of crosspoint elements of the first crossbar circuit to store the plurality of two-dimensional kernels.

[0042] In some embodiments, a method for performing the convolution further includes enabling the first plurality of crosspoint elements of the first crossbar circuit via a first plurality of selection lines of the crossbar-based device and outputting the first plurality of output signals via a first plurality of bit lines of the crossbar-based device.

[0043] In some embodiments, a method for further performing the convolution includes programming a second plurality of crosspoint elements of the second crossbar circuit to store the plurality of depth convolution kernels.

[0044] In some embodiments, a method for performing the convolution further includes enabling the second plurality of crosspoint elements of the second crossbar circuit via a second plurality of selection lines of the crossbar-based device and outputting the second plurality of output signals via a second plurality of bit lines of the crossbar-based device.

[0045] In some embodiments, programming the second plurality of crosspoint elements of the second crossbar circuit to store the plurality of depth convolution kernels includes programming a first crosspoint element to store a first element of a first kernel, wherein the first crosspoint element connects a first bit line and a first word line, and programming a second crosspoint element to store a first element of a second kernel, wherein the second crosspoint element connects a second bit line and a second word line.

[0046] In some embodiments, enabling the second plurality of crosspoint elements includes enabling the first crosspoint element and the second crosspoint element via a first selection line among the second plurality of selection lines.

[0047] This disclosure will be better understood from the detailed description provided below and from the accompanying drawings illustrating various embodiments of this disclosure. However, these drawings are for illustrative and illustrative purposes only and should not be construed as limitations of this disclosure to any particular embodiment. [Brief explanation of the drawing]

[0048] [Figure 1] This is a schematic diagram illustrating an example of a crossbar-based device according to some implementations of this disclosure. [Figure 2] This is a schematic diagram illustrating an example of a crosspoint element according to some implementations of the present disclosure. [Figure 3] This is a schematic diagram illustrating an example of a crossbar circuit according to some implementations of the present disclosure. [Figure 4A] This is a schematic diagram illustrating an example of a crossbar circuit and mapping scheme for performing depth convolution operations according to some embodiments of the present disclosure. [Figure 4B] This is a schematic diagram illustrating an example of a crossbar circuit and mapping scheme for performing depth convolution operations according to some embodiments of the present disclosure. [Figure 4C] This is a schematic diagram illustrating an example of a crossbar circuit and mapping scheme for performing depth convolution operations according to some embodiments of the present disclosure. [Figure 4D] This is a schematic diagram illustrating an example of a crossbar circuit and mapping scheme for performing depth convolution operations according to some embodiments of the present disclosure. [Figure 4E] This is a schematic diagram illustrating an example of a crossbar circuit and mapping scheme for performing depth convolution operations according to some embodiments of the present disclosure. [Figure 5A] This flowchart illustrates an example of a method for performing depth convolution using a crossbar circuit according to some embodiments of the present disclosure. [Figure 5B] This flowchart illustrates an example of a method for performing depth convolution using a crossbar circuit according to some embodiments of the present disclosure. [Figure 6A] This is an explanatory diagram illustrating an example of a convolution operation. [Figure 6B] This is an explanatory diagram illustrating an example of a convolution operation. [Figure 7A] This is an illustrative diagram illustrating an example of a data structure that can be used to store image data according to some embodiments of the disclosure. [Figure 7B] This is an illustrative diagram illustrating an example of a data structure that can be used to store image data according to some embodiments of the disclosure. [Modes for carrying out the invention]

[0049] Aspects of this disclosure provide a crossbar-based device and crossbar circuit for performing a convolution operation, and a method for performing a convolution operation using the crossbar-based device and crossbar circuit.

[0050] Crossbar circuits can be used to implement neural networks that run machine learning algorithms. Neural networks can include multiple convolution layers that perform various types of convolution operations, such as 2D convolution and depth convolution. 2D convolution can be performed by applying a single convolution kernel to the input data. More specifically, a convolution kernel can be used to scan each part of the input data, each part having the same size as the convolution kernel, to generate the convolution result. The convolution kernel and input data have the same number of channels. For example, performing a 2D convolution on a 3-channel image using a 3x3x3 kernel might require performing a scalar matrix multiplication on parts of the image having a size of 3x3x3 using the 3x3x3 kernel. As another example, performing a depth convolution on an image might require convolution of each channel of the image using each kernel, and then superimposing the convolution outputs. More specifically, performing a depth convolution on an image might require convolution of each channel of that example using a 3x3 kernel.

[0051] Images can be stored in memory as three-dimensional arrays. For example, each element of an image can be represented by a three-dimensional array, where the last dimension represents the channel dimension (e.g., an array of height values ​​(h), width values ​​(w), and channel values ​​(c)). The first element of the first channel of an image, the first element of the second channel of an image, and the first element of the third channel of an image can be represented as (h0,w0,c0), (h0,w0,c1), and (h0,w0,c2), respectively. The second element of the first channel of an image, the second element of the second channel of an image, and the second element of the third channel of an image can be represented as (h1,w0,c0), (h1,w0,c1), and (h1,w0,c2), respectively. The elements (h0,w0,c0), (h0,w0,c1), (h0,w0,c2), (h1,w0,c0), (h1,w0,c1), and (h1,w0,c2) can be stored within contiguous addresses in memory. For example, an image can be stored in memory using the data structure shown in Figures 7A and 7B.

[0052] As described above, performing 2D convolution and depth convolution on the same image requires convolution on different parts of that image using different kernels. For example, to perform a 3x3x3 portion of an image using a 3x3x3 2D convolution kernel, a conventional crossbar circuit can retrieve data items stored at addresses 710, 720, and 730, as shown in Figure 7A. As another example, to perform a depth convolution on a 3x3x3 portion of an image, a conventional crossbar circuit can perform a convolution on data items stored in addresses 711, 714, 717, 721, 724, 727, 731, 743, and 737 using a first 3x3 kernel. The conventional crossbar circuit can perform a convolution on data items stored in addresses 712, 715, 718, 722, 725, 728, 732, 735, and 738 using a second 3x3 kernel. Conventional crossbar circuits can further perform convolution of data items stored at addresses 713, 716, 719, 723, 726, 729, 733, 736, and 739 using a third 3x3 kernel. Therefore, to perform 2D convolution and depth convolution on the same image, conventional crossbar circuits may have to use different memory access patterns to retrieve image data from memory, resulting in low memory access efficiency and high (computational) power consumption.

[0053] This disclosure provides a crossbar-based device capable of performing both 2D and depth convolutions, enabling efficient memory access during 2D and depth convolution operations.

[0054] In some embodiments, a crossbar-based device may include a first crossbar circuit for performing 2D convolution and a second crossbar circuit for performing depth convolution. Each of the first and second crossbar circuits may include a crosspoint element that connects intersecting word lines and bit lines.

[0055] The first crossbar circuit can be connected to the first selection logic via a first set of selection lines. Each of the first set of selection lines can be connected to a set of crosspoint elements of the first crossbar circuit. The first selection logic can selectively enable the crosspoint elements connected to one or more of the first set of selection lines.

[0056] The second crossbar circuit can be connected to the second selection logic via a second set of selection lines. Each of the second set of selection lines can connect to multiple crosspoint elements of the second crossbar circuit that are not connected to the same word line or bit line. The second selection logic can selectively enable the crosspoint elements connected to one or more of the second set of selection lines.

[0057] The first and second crossbar circuits can be connected to word line (WL) logic via word lines. The WL logic can retrieve image data stored in a contiguous address of a memory device (e.g., an input buffer) associated with the crossbar-based device. The WL logic can further generate an input signal (e.g., a voltage signal) based on the retrieved image data and apply this input signal to the first and second crossbar circuits via word lines.

[0058] The first crossbar circuit can be programmed to store a 2D convolution kernel for performing a 2D convolution operation. For example, the crosspoint elements of the first crossbar circuit can be programmed to conductance values ​​representing a 2D convolution kernel. In response to an input signal, the first crossbar circuit can output a current signal representing the convolution result of the image and the 2D convolution kernel.

[0059] The second crossbar circuit can be programmed to store depth convolution kernels for performing depth convolution operations. For example, the crosspoint elements of the second crossbar circuit can be programmed to conductance values ​​representing depth convolution kernels. In response to an input signal, the second crossbar circuit can output a current signal representing the convolution result of the depth kernel. Thus, a crossbar-based device can perform both 2D convolution and depth convolution using the same memory access pattern, which enables efficient memory use in neural networks incorporating crossbar-based devices.

[0060] Figure 1 is a schematic diagram illustrating an example of a device 100 for performing a convolution operation according to some embodiments of this disclosure. Here, the device 100 may also be referred to as a crossbar-based device. The device 100 may be a neural processing unit (NPU) or part of an NPU for executing machine learning algorithms.

[0061] As shown in the figure, the device 100 may include a first crossbar circuit 101 and a second crossbar circuit 102 for performing convolution operations. Each of the first crossbar circuit 101 and the second crossbar circuit 102 may include one or more crossbar arrays. Each crossbar array may include one or more intersecting wires, such as word lines (WL) 111 and bit lines (BL) 113. Furthermore, a crossbar array may include one or more crosspoint elements 120 that connect one or more intersections between word lines and bit lines. A crosspoint element connecting the j-th word line and the k-th bit line may be called a crosspoint element 120(j,k). Each crosspoint element may be a programmed resistor such as a memristor, a phase-change memory (PCM) element, a floating gate element, a spintronic element, a resistive random-access memory (RRAM), or a static random-access memory (SRAM). In some embodiments, the crosspoint element 120 may be a 1-transistor-1-memristor (1T1M) circuit structure, a 1-selector-1-resistor (1S1R) structure, a 2-resistor (2R) structure, etc., or may include both or either of these. In some embodiments, one or more crosspoint elements 120 may include crosspoint elements 200 as described in relation to Figure 2.

[0062] As illustrated in Figure 1, the first crossbar circuit 101 and the second crossbar circuit can be connected to the word line (WL) logic 103 via word lines 111 (e.g., WL 111-1, 111-2, 111-3, 111-4, etc.) and can also be connected to the programming circuit 104 and the sensing circuit 105 via bit lines 113. The WL logic 103 can include any suitable components for converting input data into input signals applied to the crossbar circuit 101 and / or the crossbar circuit 102. Each of the input signals can be a voltage signal, a current signal, etc. In some embodiments, the WL logic 103 can be one or more digital-to-analog converters (DACs) that can convert the input data into analog signals.

[0063] The programming circuit 104 can program the crosspoint element 120 to an appropriate conductance value. For example, programming a crosspoint element may require the application of an appropriate voltage or current signal across that crosspoint element. The resistance of each crosspoint element can be electrically switched between a high-resistance state and a low-resistance state. Setting a crosspoint element may require switching the resistance of the crosspoint from a high-resistance state to a low-resistance state. Resetting a crosspoint element may require switching the resistance of the crosspoint from a low-resistance state to a high-resistance state.

[0064] The first crossbar circuit 101 can be connected to the first selection logic 106 via a selection line 140. The second crossbar circuit 102 can be connected to the second selection logic 107 via a selection line 130. Each of the selection lines 130 and 140 may include any suitable conductive material. In some embodiments, each selection line 130 and 140 may be a metal wire, or both, including one or the other. While a suitable voltage (sometimes referred to here as a selection voltage) is applied to a given selection line 130, the crosspoint elements connected to that selection line 130 can be enabled. The first selection logic 106 can selectively enable one or more crosspoint elements of the first crossbar circuit 101 via one or more selection lines 140 (for example, by applying a selection voltage to one or more selection lines 140). The second selection logic 107 can selectively enable one or more crosspoint elements of the second crossbar circuit 102 via one or more selection lines 130 (for example, by applying a selection voltage to one or more selection lines 130).

[0065] The sensing circuit 105 can generate an output signal based on the current flowing through the bit line 113. In some embodiments, the sensing circuit 105 can include any suitable components for converting the current into a digital output. For example, the sensing circuit 105 can include a plurality of analog-to-digital converters (ADCs) (not shown). Each ADC can convert the current flowing through its respective bit line into a digital output. In some embodiments, the input signal can include a voltage signal V. The output signal can include a current signal I. The relationship between the input signal and the output signal can be expressed as I=VG, where G represents the conductance value of the crosspoint element 120. Thus, the input signal is weighted by the conductance of each crosspoint element according to Ohm's law. The weighted currents are output through each bit line and can be integrated according to Kirchhoff's current law.

[0066] Each of the crossbar circuits 101 and 102 can be configured to perform vector-matrix multiplication (VMM). The VMM operation can be expressed as Y=XA, where Y, X, and A each represent their respective matrices. More specifically, for example, the input vector X can be mapped to the input voltage V of the crossbar circuits 101 and / or 102. Matrix A can be mapped to the conductance value G. The output current I can be read out and mapped back to the output result Y.

[0067] In some embodiments, the first crossbar circuit 101 and the second crossbar circuit 102 can perform different types of convolution operations. For example, the first crossbar circuit 101 can perform a 2D convolution operation, while the second crossbar circuit 102 can perform a depth convolution.

[0068] Performing a 2D convolution on input data may require applying a single convolution kernel to that input data. The convolution kernel can have a specific size defined by multiple dimensions (e.g., width, height, channels, etc.). The convolution kernel can be applied to a portion of the input data that has the same size as the convolution kernel to produce an output. The output can then be mapped to the element of the convolution result located at the position corresponding to that portion of the input data.

[0069] For example, as shown in Figure 6A, a 2D convolution can be performed on image 610 using kernel 620 to generate output image 630. Image 610 may contain a first channel 610a, a second channel 610b, and a third channel 610c. The size of image 610 can be defined by its width (w), height (h), and channels (c). As an example, the size of image 610 may be 7 × 7 × 3. The size of kernel 620 may be 3 × 3 × 3. Kernel 620 can be applied to the first portion 611 of image 610. As shown, the size of the first portion 611 is the same as the size of kernel 620. The first element 631 of the output image can be obtained by scalar multiplication of the first portion 611 by kernel 620. Multiple 3 × 3 × 3 portion parts of image 610 can be scanned using kernel 620 to generate elements of output image 630. In some embodiments, the kernel 620 can be applied to a 5x5x1 portion of the image 610 to generate a 5x5x1 output image 630.

[0070] The elements of image 610 can be defined by a three-dimensional array of height, width, and channel values. For example, as shown in Figure 6A, the first part 611 of image 610 can be considered as images of three channels 611a, 611b, and 611c. Each of the images of channels 611a, 611b, and 611c can contain nine elements defined by various width (w) values, height (h) values, and channel (c) values, as shown in Figure 6A. Image 610 can be stored in a memory device as a three-dimensional array using the data structure illustrated in Figures 7A-7B.

[0071] Performing depth convolution on input data may require convolution of each channel of the input data using the corresponding kernel, and then superimposing the convolutional outputs. For example, performing depth convolution on image 610 may require convolution of each channel of image 610 using kernels 621, 623, and 625, respectively. Kernels 621, 623, and 625 can each correspond to a channel of kernel 620 in Figure 6A. Convolution of channel 611a of image 610 using kernel 621 may require element-wise multiplication between the elements of channel 611a of image 610 and the elements of kernel 621 at the same location. Convolution of the first channel 611a of image 610 using kernel 621 can produce output 641. Convolution of the second channel 611b of image 610 using kernel 623 can produce output 643. Convolution of the third channel 611c of image 610 using kernel 625 can produce output 645. Outputs 641, 643, and 645 can be superimposed as output 640. Then, point-by-point convolution can be performed on output 640 using 1×1×3 kernel 650 to produce output image 630.

[0072] Referring back to Figure 1, the WL logic 103 can retrieve input data to be processed by the crossbar circuits 101 and / or 102 from a storage device associated with the device 100. The storage device can be an input buffer in some embodiments. The WL logic 103 can convert the input data into a vector and, further, generate an input signal representing that vector. The input signal can be applied to the crossbar circuits 101 and / or 102 via one or more word lines 111.

[0073] In some embodiments, input signals applied to the first and second crossbar circuits via 111-1, 111-2, 111-3, ..., 111-n can be made to correspond to data stored in a contiguous address range of the memory. For example, to perform a convolution on the first portion 611 of image 610 in Figures 6A-6B, the WL logic 103 can retrieve image data of the first portion 611 of image 610 stored in addresses 710, 720, and / or 730 of the memory as shown in Figure 7A.

[0074] More specifically, for example, the first data item stored in the first address 711 of the memory device can be applied to the first and second crossbar circuits via word line 111a. The second data item stored in the second address 712 of the memory device can be associated with a second input signal applied to the first and second crossbar circuits via word line 111b. The third data item stored in the third address 713 of the memory device can be associated with a third input signal applied to the first and second crossbar circuits via word line 111c. The i-th data item stored in the i-th address of the memory device can be applied to the first and second crossbar circuits via 111i. The first address, the second address, ..., the i-th address, etc., can be consecutive.

[0075] The programming circuit 104 can program the first crossbar circuit 101 to store 2D convolution kernels for performing 2D convolution operations. For example, a 2D convolution kernel can be converted into a vector and mapped to a plurality of crosspoint elements of the first crossbar circuit connected to a given bit line. In particular, the conductance values ​​of the crosspoint elements can be programmed to represent values ​​representing the 2D convolution kernel. The first crossbar circuit 101 can output a current signal representing the convolution of the image and the 2D convolution kernel via a given bit line in response to an input signal. In some embodiments, the first crossbar circuit 101 can store a plurality of 2D convolution kernels by mapping each of the 2D convolution kernels to a crosspoint element connected to each bit line. The first crossbar circuit 101 can output a plurality of output signals (e.g., current signals) representing the convolution result via a plurality of first bit lines 113a.

[0076] The programming circuit 104 can program a second crossbar circuit 102 that can be programmed to store depth convolution kernels for performing depth convolution operations. For example, the crosspoint elements of the second crossbar circuit can be programmed to conductance values ​​representing depth convolution kernels. In response to an input signal, the second crossbar circuit can output current signals representing the convolution result of the depth convolution operation via a second set of bit lines 113b. In some embodiments, as described with reference to Figure 3-4D, the crossbar circuit 102 may include one or more crossbar circuits.

[0077] Figure 2 is a schematic diagram illustrating an example of a crosspoint element 200 according to some embodiments of the present disclosure. As shown, the crosspoint element 200 can be connected to a bit line (BL) 211, a selection line (SEL) 213, and a word line (WL) 215. The bit line 211, the selection line 213, and the word line 215 can be bit line 113, selection line 130 or 140, and word line 111, respectively, as described in relation to Figure 1.

[0078] The crosspoint element 200 can include an RRAM element 201 and a transistor 203. The transistor is a three-terminal element, each of which can be marked as gate (G), source (S), and drain (D). Transistor 203 can be connected in series with the RRAM element 201. As shown in Figure 2, the first electrode of the RRAM element 201 can be connected to the drain of transistor 203. The second electrode of the RRAM element 201 can be connected to the bit line 211. The source of transistor 203 can be connected to the word line 215. The gate of transistor 203 can be connected to the selection line 213. The RRAM element 201 can include one or more RRAM elements, as described below in relation to Figures 3A-5B. The crosspoint element 200 can also be said to have a 1-transistor-1-resistor (1T1R) configuration. Transistor 203 can act as a selector and current controller, and can set current compliance for the RRAM element 201 during programming. The gate voltage of transistor 203 can be used to set the current compliance with respect to the crosspoint element 200 during programming, and thus the conductance and analog behavior of the crosspoint element 200 can be controlled. For example, when setting the crosspoint element 200 from a high-resistance state to a low-resistance state, a set signal (e.g., a voltage signal, a current signal) can be provided via the bit line (BL) 211. Another voltage, also called a selection voltage or gate voltage, can be applied to the gate of the transistor via the selection line (SEL) 213 to open the gate and set the current compliance, while the word line (WL) 215 can be set to ground. When resetting the crosspoint element 200 from a low-resistance state to a high-resistance state, the gate voltage can be applied to the gate of transistor 203 via the selection line 213 to open the gate of the transistor. On the other hand, a reset signal can be sent to the RRAM element 201 via the word line 215, while the bit line 211 can be set to ground.

[0079] Figure 3 is an illustrative diagram of an example crossbar circuit 300 capable of performing depth convolution operations. The crossbar circuit 300 can be the crossbar circuit 102 as described in relation to Figure 1.

[0080] As shown in the figure, the crossbar circuit 300 may include word lines 111, bit lines 113, crosspoint elements 120, and selection lines 130 (for example, selection lines 130-1, 130-2, etc.). In some embodiments, the number of selection lines may be n-m+1, where n and m refer to the number of word lines 111 and the number of bit lines 113, respectively.

[0081] In some embodiments, each selection line 130 can electrically connect multiple crosspoint elements (for example, crosspoint elements in different rows and different columns) that do not share any bit lines or word lines. In some embodiments, the selection line 130 is not parallel to the bit line 113 or the word line 111. For example, selection line 130-1 can connect the first crosspoint element 120(1,1), the second crosspoint element 120(2,2), the third crosspoint element 120(3,3), etc. The first crosspoint element 120(1,1) can connect the first word line 111-1 and the first bit line 113-1. The second crosspoint element 120(2,2) can connect the second word line 111-2 and the second bit line 113-2. The third crosspoint element 120(3,3) can connect the third word line 111-3 and the third bit line 113-3. Therefore, the first, second, and third crosspoint elements are not connected to the same word line or the same bit line. As another example, the selection line 130-2 can connect the crosspoint elements 120(2,1), 120(3,2), ..., 120((m+1),m).

[0082] While an appropriate voltage is applied to a given selection line 130, the crosspoint elements connected to that selection line 130 can be activated. Crosspoint elements that can be activated by one or more of the selection lines 130 are also referred to herein as “active cells”. Crosspoint elements that cannot be activated by the selection lines 130 are also referred to herein as “dummy cells”. Dummy cells do not function when the crossbar circuit 300 performs a depth convolution operation. In one implementation, dummy cells are not connected to the selection lines. In another implementation, dummy cells are connected to one or more selection lines (not shown) but do not become active during the convolution operation. In a further implementation, the crossbar circuit 300 does not include dummy cells. As illustrated, the active cells can form a parallelogram shape in some embodiments.

[0083] To perform a depth convolution operation on input data (for example, one or more images), the crossbar circuit 300 can compute a matrix multiplication of one or more matrices corresponding to the input data and one or more matrices corresponding to the depth convolution kernel. The input data may have multiple channels. The depth convolution kernel can be divided into multiple two-dimensional kernels. Each of the two-dimensional kernels can be used for the convolution of each channel of the input data. The elements of the kernel can be mapped to the crosspoint elements of the crossbar circuit 300 by programming the conductance of the crosspoint elements to values ​​corresponding to the values ​​of those elements in the kernel. The crosspoint elements can be programmed to store the depth convolution kernel using one or more mapping schemes described below in relation to Figures 4A-4E.

[0084] For example, the convolution of the first channel of input data using a kernel may require the calculation of a matrix multiplication between a first matrix representing the first channel of the first image and a second matrix representing the kernel. The crossbar circuit 300 can be programmed so that the conductance values ​​of the crosspoint elements can represent the values ​​of the elements of the kernel (for example, by programming the crosspoint elements using programming signals corresponding to the conductance values). The first matrix can be converted into multiple vectors representing parts of the first matrix. The crossbar circuit 300 can receive multiple input signals representing those vectors. When the input signals are applied to the crossbar circuit 300 via word lines, Ohm's law dictates that the current through each crosspoint element is the product of its conductance and the corresponding voltage applied to the word line connected to that crosspoint element. Kirchhoff's current law dictates that the currents from each crosspoint element connected to the same bit line are multiplied. The output current from each bit line represents the result of the multiplication-integration operation, and the convolution of the first channel of the input data with the kernel.

[0085] The crossbar circuit 300 can perform multiple depth convolution operations in parallel using the kernel mapping scheme described below in relation to Figures 4A-4E.

[0086] Figure 4A illustrates an example of a crossbar circuit 400a for performing a depth convolution operation according to some embodiments of the present disclosure. The crossbar circuit 400a may be the crossbar circuit 300 as described in relation to Figure 3, or both, including it.

[0087] As shown in the diagram, the crossbar circuit 400a may include multiple crosspoint elements 420 (e.g., 420(1,1), 420(2,2), etc.) that connect multiple word lines (e.g., word lines 401, 403, 405) and multiple bit lines (bit lines 407-1, 407-2, 407-3, ..., 407-m). Crosspoint element 420(j,k) may refer to a crosspoint element that connects the j-th word line and the k-th bit line. The crossbar circuit 400a may further include selection lines to selectively enable one or more of the crosspoint elements 420. For example, a first selection line 430-1 may connect the first group of crosspoint elements of the crossbar circuit 400a (e.g., crosspoint elements 420(1,1), 420(2,2), 420(3,3), ..., 420(m,m)). The second selection line 430-2 can connect to the second group of crosspoint elements of the crossbar circuit 400a (for example, crosspoint elements 420(m+1, 1), 420(m+2, 2), 420(m+3, 3), ..., 420(m+m, m)). The i-th selection line 430-i can connect to the i-th group of crosspoint elements of the crossbar circuit 400a (for example, crosspoint elements 420(2m+i-1, 1), 420(2m+i, 2), 420(2m+i+1, 3), ..., 420(3m+i-1, m)). The crosspoint elements, word lines, bit lines, and selection lines shown in Figure 4A may be the same as their corresponding parts in Figure 1-3.

[0088] The crossbar circuit 400a can store multiple kernels and perform depth convolution operations in parallel. Each kernel can be used to convolve a portion of each channel of the input data. Elements of kernels at the same position (for example, a position defined by the same width and height) can be mapped to crosspoint elements connected to the same selection line. Elements of a given kernel can be mapped to multiple crosspoint elements connected to the same bit line. For example, the first element of each kernel can be mapped to a crosspoint element connected to a first selection line. More specifically, for example, the first element of the first kernel, the first element of the second kernel, the first element of the third kernel, ..., and the first element of the m-th kernel can be mapped to crosspoint elements 420(1,1), 420(2,2), 420(3,3), ..., and 420(m,m), respectively. As another example, the second element of each kernel can be mapped to a crosspoint element connected to a second selection line 430-2. More specifically, for example, the second element of the first kernel, the second element of the second kernel, the second element of the third kernel, ..., and the second element of the m-th kernel can be mapped to the crosspoint elements 420(m+1, 1), 420(m+2, 2), 420(m+3, 3), ..., 420(m+m, m), respectively. As another example, the i-th element of each kernel can be mapped to the crosspoint elements connected to the i-th selected line 430-i (for example, crosspoint elements 420(2m+i-1, 1), 420(2m+i, 2), 420(2m+i+1, 3), ..., 420(3m+i-1, m)).

[0089] In one implementation, word lines 401-m and 403-1 are adjacent to each other. For example, selection lines 430-1 and 430-2 may correspond to selection lines 434-1 and 434-2 in Figure 4C, respectively. In another implementation, word lines 401-m and 403-1 are not adjacent to each other. One or more other word lines (not shown) are positioned between word lines 401-m and 403-1. For example, selection lines 430-1 and 430-2 may correspond to selection lines 431-1 and 432-1 in Figure 4B, respectively. Similarly, word lines 403-m and 405-1 may or may not be adjacent to each other.

[0090] The crossbar circuit can perform m depth convolution operations in parallel. Each depth convolution operation can generate a convolution of a kernel having i elements from a portion of the input data. For example, a first selection line, a second selection line, ..., and the i-th selection line can be selected (for example, by applying an appropriate voltage to each of the selection lines). The crosspoint elements connected to the first selection line, the second selection line, ..., and the i-th selection line can therefore be enabled. Input signals representing the input data to be convolved can be applied to the enabled crosspoint elements via word lines connected to the enabled crosspoint elements. For example, input signals 411-1, 411-2, 411-3, ..., 411-m can be applied to the first group of crosspoint elements via word lines 401-1, 401-2, 401-3, ..., 401-m, respectively. Input signals 413-1, 413-2, 413-3, ..., 413-m can be applied to a second group of crosspoint elements via word lines 403-1, 403-2, 403-3, ..., 403-m, respectively. Input signals 415-1, 415-2, 415-3, ..., 415-m can be applied to a third group of crosspoint elements via word lines 405-1, 405-2, 405-3, ..., 405-m, respectively. Input signals 411-1, 411-2, ..., 411-m can correspond to portions of input data corresponding to a first width value (e.g., w0). Input signals 413-1, 413-2, ..., 413-m can correspond to a second portion of input data corresponding to a second width value (e.g., w1). Input signals 415-1, 415-2, ..., 415-m can correspond to the third portion of the input data corresponding to the third width value (e.g., w2). Input signals 411-1, 413-1, 415-1 can correspond to the first channel of the input data (e.g., c0). Input signals 411-2, 413-2, 415-2 can correspond to the second channel of the input data (e.g., c1).Input signals 411-3, 413-3, and 415-3 can correspond to a third channel (e.g., c2) of the input data. For example, input signals 411, 413, and 415 can represent data items stored in addresses 710, 720, and 730 in Figure 7, respectively. More specifically, input signals 411-1, 411-2, ..., 411-m can represent data items stored in consecutive addresses 711, 713, ..., 719 of the memory device.

[0091] The integrated current output through bit lines 417 (for example, bit lines 417-1, 417-2, 417-3, ..., and 417-m) can represent the convolution of the channels of the input data and the kernels mapped to the crosspoint elements connected to those bit lines. For example, the first, second, third, ..., and m output signals can be output through bit lines 417-1, 417-2, 417-3, ..., and 417-m, respectively. The first output signal can represent the convolution of the first channel of the input data and the first kernel. The second output signal can represent the convolution of the second channel of the input data and the second kernel.

[0092] Referring to Figure 4B, the crossbar circuit 400b may include a first group of selection lines 431 connecting a first plurality of crosspoint elements 421, a second group of selection lines 432, ... connecting a second plurality of crosspoint elements 422, ... and an i-th group of selection lines 433 connecting an i-th group of crosspoint elements 423. Each of the selection lines 431, 432, and 433 may be a selection line 430 as described in relation to Figure 4A. Each of the plurality of crosspoint elements 421, 422, and 423 may be a crosspoint element 420 as described in relation to Figure 4A. Each group of selection lines may include a certain number of selection lines. The number of selection lines in different groups of selection lines (for example, groups of selection lines 431 and 432) may be the same or different.

[0093] The crossbar circuit 400b can be configured to perform multiple depth convolution operations in parallel using multiple kernels, and to perform depth convolution operations sequentially using multiple groups of kernels. For example, the first element of each of the first multiple kernels can be mapped to a crosspoint element connected to selection line 431-1. The second element of each of the first multiple kernels can be mapped to a crosspoint element connected to selection line 432-1. The i-th element of each of the first multiple kernels can be mapped to a crosspoint element connected to selection line 433-1. The first, second, ..., i-th elements of each of the second multiple kernels can be mapped to crosspoint elements connected to selection lines 431-2, 432-2, ..., and 433-2, respectively. The first, second, ..., i-th elements of each of the nth plurality of kernels can be mapped to crosspoint elements connected to selected lines 431-N, 432-N, ..., and 433-N, respectively. Each kernel of the first plurality of kernels, the second plurality of kernels, and the third plurality of kernels can be a depth convolution kernel used to perform a depth convolution operation.

[0094] To perform a depth convolution operation using the first set of kernels, the first selection lines of each group of selection lines (e.g., selection lines 431-1, 432-1, ..., 433-1) can be selected, and the other selection lines can be deselected. For example, an appropriate selection voltage can be applied to each of the first selection lines of the group of selection lines. Thus, the crosspoint elements connected to the first selection lines of the group of selection lines can be enabled. The first input signals representing the first input data to be convolved using the first set of kernels can be applied to the enabled crosspoint elements via word lines (not shown) connected to those enabled crosspoint elements.

[0095] As another example, to perform a depth convolution operation using a second set of kernels, one can select the second selection line for each of the group of selection lines (for example, selection lines 431-2, 432-2, ..., 433-2). Thus, the crosspoint elements connected to selection lines 431-2, 432-2, ..., 433-2 can be enabled. The second input signal representing the second input data to be convolved using the second set of kernels can be applied to the enabled crosspoint elements via word lines (not shown) connected to those enabled crosspoint elements.

[0096] Figure 4C illustrates an example of a crossbar circuit 400c according to some embodiments of the present disclosure.

[0097] As illustrated, the crossbar circuit 400c may include groups 424, 425, ..., 426 of crosspoint elements and selection logic 460a. Groups 424, 425, ..., 426 of crosspoint elements may be connected to selection logic 460a via selection lines 434, 435, and 436, respectively. Each of the selection lines 434, 435, and 436 may be the selection line 430 in Figure 4A, or both, or include it. Each crosspoint element in groups 424, 425, ..., 426 may include multiple crosspoint elements 420 as described above in relation to Figure 4A. The crosspoint elements in the crossbar array 424 may be connected to a first portion 461 of selection logic 460a via selection line 434. Crosspoint elements in the crossbar array 425 can be connected to a second portion 463 of the selection logic 460a via selection line 435. Crosspoint elements in the crossbar array 426 can be connected to a third portion 465 of the selection logic 460a via selection line 436. The selection logic 460a may include selection registers 461-1, 461-2, 461-3, 461-4, 461-5, 461-6, ..., 461-N, 463-1, 463-2, 463-3, 463-4, 463-5, 463-6, ..., 463-N, 465-1, 465-2, 465-3, 465-4, 465-5, 465-6, ..., and 465-N. As shown in the diagram, each of the selection lines 434, 435, and 436 can be connected to the respective selection registers of the selection logic 460a. Crosspoint elements connected to the same selection line can be enabled and used simultaneously for the execution of convolution operations. Each of the selection lines 434, 435, and 436 can be connected to a specific number of crosspoint elements. The number of crosspoint elements connected to one of the selection lines of the crossbar circuit 400c can correspond to the number of convolution operations that the crossbar circuit 400c can execute in parallel.

[0098] A depth convolution kernel can be mapped to one or more crosspoint elements of the crossbar circuit 400c to perform a depth convolution operation. For example, as described in relation to Figure 4B, a first plurality of kernels, a second plurality of kernels, and a third plurality of kernels can be mapped to crosspoint elements 424, 425, and 426, respectively. To perform a depth convolution operation using the first plurality of kernels, the crossbar circuit 400c can enable crosspoint element 424 via selection line 434. A first input signal representing the first input data to be convolved using the first plurality of kernels can be applied to the enabled crosspoint element 424. Each of these input signals can be a voltage signal representing a portion of the first input data. The outputs generated via bit lines 417-1, 417-2, ..., 417-m can represent the convolution result of the first input data and the first plurality of kernels.

[0099] To perform a depth convolution operation using a second set of kernels, the crossbar circuit 400c can enable the crosspoint element 425 via the selection line 435. A second input signal representing the second input data to be convolved using the second set of kernels can be applied to the enabled crosspoint element 425. Each of the second input signals can be a voltage signal representing a portion of the second input data. The outputs generated via bit lines 417-1, 417-2, ..., 417-m can represent the convolution result of the second input data and the second set of kernels.

[0100] The crossbar circuit 400c can perform depth convolution operations using kernels of any appropriate size. For example, each of the crosspoint element groups 424, 425, and 426 can store eight kernels, and these eight kernels can be used to perform parallel convolution of input data through eight different channels. Each of the crosspoint element groups 424, 425, and 426 is sometimes called a crossbar array. The crossbar circuit 400c can perform depth convolution operations on input data for three channels. Three depth convolution kernels can be mapped to the crossbar circuit 400c, and convolutions can be performed on the input data for the first channel, the second channel, and the third channel, respectively. The size of the depth convolution kernels (for example, defined by the width dimension and the height dimension) can be the same. For example, the first kernel, the second kernel, and the third kernel can each be a 3x3 kernel with a width of 3 and a height of 3.

[0101] In some embodiments, the crossbar circuit 400c can perform a depth convolution operation on input data using a 3x3 depth kernel having a width of 3 and a height of 3. For example, the crossbar circuit 400c can store a first 3x3 kernel, a second 3x3 kernel, and a third 3x3 kernel by setting the conductance of the crosspoint elements of the crossbar circuit 400c to values ​​representing the elements of those kernels. For example, the first element of the first kernel, the first element of the second kernel, and the first element of the third kernel can be mapped to the crosspoint elements 420(1,1), 420(2,2), and 420(3,3), respectively. The first element of the first kernel, the first element of the second kernel, and the first element of the third kernel can be defined by a first width value (e.g., w0) and a first height value (e.g., h0). The second element of the first kernel, the second element of the second kernel, and the second element of the third kernel can be mapped to the crosspoint elements 420(4,1), 420(5,2), and 420(6,3), respectively. The second element of the first kernel, the second element of the second kernel, and the second element of the third kernel can be defined by a first width value (e.g., w0) and a second height value (e.g., h1). The third element of the first kernel can be mapped to the crosspoint element 420(7,1). The third element of the second kernel can be mapped to the crosspoint element 420(8,2). The third element of the third kernel can be mapped to the crosspoint element 420(9,3). The third element of the first kernel, the third element of the second kernel, and the third element of the third kernel can be defined by a first width (e.g., w0) and a third height (e.g., h2).

[0102] The remaining elements of the first kernel, the second kernel, and the third kernel can be mapped to the crossbar circuit 400c in a similar manner. For example, the i-th element of the first kernel, the i-th element of the second kernel, and the i-th element of the third kernel can be mapped to three crosspoint elements connected by the i-th selected line. The ninth element of the first kernel (w2h2c0) can be mapped to the crosspoint element 420(27,1) (not shown). The ninth element of the second kernel (w2h2c1) can be mapped to the crosspoint element 420(28,2) (not shown). The ninth element of the third kernel (w2h2c2) can be mapped to the crosspoint element 420(29,3) (not shown).

[0103] To perform a depth convolution operation, multiple input signals can be applied to multiple word lines (not shown). Each word line can be connected to an enabled crosspoint element. For example, a first input signal, a second input signal, a third input signal, a fourth input signal, ..., and a ninth input signal can be applied to the first word line, the second word line, the third word line, the fourth word line, ..., and the ninth word line of the crossbar circuit, respectively. The first input signal may represent the first element of the first channel of the image, defined by a first width (w0) and a first height (h0). The second input signal may represent the first element of the second channel of the image, defined by a first width and a first height. The third input signal may represent the first element of the third channel of the image, defined by a first width and a first height. In some embodiments, the first input signal, the second input signal, ..., and the ninth input signal can each represent a data item stored in consecutive addresses 711, 712, 713, 714, 715, 716, 717, 718, and 719, as illustrated in Figure 7.

[0104] Since the first kernel is mapped to a crosspoint element connected to the first bit line, the integrated current output through the first bit line 417-1 can represent the convolution of the first channel of the input data and the first kernel. Similarly, the integrated current output through the second bit line 417-2 can represent the convolution of the second channel of the input data and the second kernel. The integrated current output through the third bit line can represent the convolution of the third channel of the image and the third kernel.

[0105] Figures 4D and 4E illustrate an example of a kernel mapping scheme for performing depth convolution by a crossbar circuit 400d according to some embodiments of the present disclosure.

[0106] As illustrated, the crossbar circuit 400d can include a crossbar array 450b and a selection logic 460b. The crossbar array can include one or more crossbar arrays 300, as previously mentioned in relation to Figure 3. Selection lines 130-1, 130-2, 130-3, ..., 130-8 can be connected to the selection register 462 and enabled simultaneously. In some embodiments, the crossbar circuit 400d can include nine or more crossbar arrays. Thus, the crossbar circuit 400d can include 72 selection lines. The 72 selection lines can be divided into nine groups. Each group of selection lines can include eight selection lines. Selection lines within the same group can be connected to a shared selection register and selected simultaneously. Therefore, crosspoint elements connected by selection lines in the same group can be enabled simultaneously.

[0107] As previously mentioned in relation to Figure 4C, in order to perform depth convolution on a 3-channel image using 3x3 kernels, the crossbar circuit 400d can be programmed to store the crosspoint elements. The crossbar circuit 400c can store 64 sets of 1-channel 3x3 kernels, and it is possible to run 8 of them simultaneously using 8 different channels.

[0108] For example, as shown in Figure 4E, the first, second, and third elements of the first kernel can be mapped to the crosspoint elements 120(1,1), 120(9,1), and 120(17,1), respectively. The first, second, and third elements of the second kernel can be mapped to the crosspoint elements 120(2,2), 120(10,2), and 120(18,2), respectively. The first, second, and third elements of the third kernel can be mapped to the crosspoint elements 120(3,3), 120(11,3), and 120(19,3), respectively. The remaining elements of the first, second, and third kernels can be mapped to the crossbar circuit 400c in a similar manner. The ninth element of the first kernel (w2h2c0) can be mapped to the cross point element 120(65,1) (not shown). The ninth element of the second kernel (w2h2c1) can be mapped to the cross point element 120(66,2). The ninth element of the third kernel (w2h2c2) can be mapped to the cross point element 120(67,3).

[0109] If the number of channels in the depth convolution is less than 8, a data gap may be required in the input buffer. The number of gaps can be the number of bit lines (8) minus the number of channels.

[0110] Figures 5A and 5B are flowcharts illustrating examples of methods 500 and 550 for performing depth convolution using crossbar circuits according to some embodiments of the present disclosure. The crossbar circuits may be crossbar circuits 102, 300, 400a, 400b, 400c, and / or 400d, or both, including them.

[0111] If multiple depth convolution kernels can be mapped to multiple crosspoint elements of a crossbar circuit, process 500 can begin from block 505. The crosspoint elements can be connected to multiple selection lines. Each selection line can connect to crosspoint elements connected to different word lines and different bit lines of the crossbar circuit. For example, these crosspoint elements may include those connected to selection lines 430-1, 430-2, ..., 430-i, as illustrated in relation to Figure 4A. Another example may include those connected to selection lines 431-1, 432-1, ..., 433-1, as illustrated in relation to Figure 4B. A further example may include those connected to selection line 434, as illustrated in relation to Figure 4C.

[0112] In some embodiments, each first element of the depth convolution kernel can be mapped to a first group of crosspoint elements connected to a first selection line (for example, the crosspoint element connected to selection line 430-1 in Figure 4A, and the crosspoint element connected to selection line 431-1 in Figure 4B). Each second element of the depth convolution kernel can be mapped to a second group of crosspoint elements connected to a second selection line (for example, the crosspoint element connected to selection line 430-2 in Figure 4A, the crosspoint element connected to selection line 432-1 in Figure 4B, and the crosspoint element connected to selection line 434-2 in Figure 4C). Each i-th element of the depth convolution kernel can be mapped to an i-th group of crosspoint elements connected to an i-th selection line (for example, the crosspoint element connected to selection line 430-i in Figure 4A, the crosspoint element connected to selection line 433-1 in Figure 4B, and the crosspoint element connected to selection line 434-N in Figure 4C). In some embodiments, each kernel can be a k×k kernel.

[0113] A depth-direction convolution kernel can be mapped to a crosspoint element by programming the conductance value of the crosspoint element to the value representing the element of the kernel. For example, the first element of the first kernel can be mapped to the first crosspoint element by programming the first crosspoint element to the first conductance value representing the first element of the first kernel. The first crosspoint element can connect the first bit line and the first word line of a crossbar circuit. Another example is that the first element of the second kernel can be mapped to the second crosspoint element by programming the second crosspoint element to the second conductance value representing the first element of the second kernel. The second crosspoint element can connect the second bit line and the second word line of a crossbar circuit. The first element of the first kernel and the first element of the second kernel correspond to a first height value and a first width value. As yet another example, the second element of the first kernel can be mapped to a third crosspoint element by programming the third crosspoint element to a third conductance value representing the second element of the first kernel. The third crosspoint element can connect the second word line and the first bit line. As yet another example, the second element of the second kernel can be mapped to a fourth crosspoint element by programming the fourth crosspoint element to a fourth conductance value representing the second element of the second kernel. The fourth crosspoint element can connect the third word line and the second bit line of the crossbar circuit. The second element of the first kernel and the second element of the second kernel can correspond to a second height value and a first width value.

[0114] In block 510, multiple crosspoint elements can be enabled via multiple selection lines. For example, a first group of crosspoint elements can be enabled via a first selection line. The first group of crosspoint elements may include a first crosspoint element and a second crosspoint element. As another example, a second group of crosspoint elements can be enabled via a second selection line. The second group of crosspoint elements may include a third crosspoint element and a fourth crosspoint element. In some embodiments, enabling a crosspoint element may require applying an appropriate selection voltage to each of the multiple selection lines.

[0115] In block 515, multiple input signals can be applied to the crossbar circuit. These input signals can be applied to the crossbar circuit via multiple word lines (for example, word lines 401, 403, and 405 as described in relation to Figure 4A) connected to the enabled crosspoint element. These input signals can represent input data that is convolved using a kernel mapped to the crossbar circuit. For example, the multiple input signals may include a first input signal representing a first portion of a first channel of input data, and a second input signal representing a second portion of a second channel of input data. The first and second input signals can be applied to the crossbar circuit via the first and second word lines, respectively. In some embodiments, the input data may include one or more images.

[0116] In block 520, the crossbar circuit can generate multiple output signals representing the convolution result of the input data and the kernel via multiple bit lines of the crossbar circuit. These output signals may be output signals 417-1, 417-2, ..., 417-m, output via bit lines 407-1, 407-2, ..., 407-m, as described in relation to Figure 4A. For example, a first output signal (e.g., output signal 417-1 in Figure 4A) representing a first portion of the input data and a first convolution of a first kernel can be output via the first bit line. As another example, a second output signal (e.g., output signal 417-2 in Figure 4A) representing a second portion of the input data and a second convolution of a second kernel can be output via the second bit line. The first portion of the input data and the second portion of the input data may correspond to a first channel and a second channel of the input data, respectively.

[0117] If the first set of kernels can be mapped to the first set of crosspoint elements of the crossbar circuit, process 550 can begin from 555. The first set of crosspoint elements can be connected to a first group of selection lines. Each of the first group of selection lines can connect to a set of crosspoint elements that do not share a bit line or word line. As an example, as previously stated in relation to Figure 4B, the first set of kernels can be mapped to crosspoint elements connected to selection lines 431-1, 432-1, ..., and 433-1. As another example, as previously stated in relation to Figure 4C, the first set of kernels can be mapped to crosspoint elements connected to selection lines 434-1, 434-2, ..., and 434-N.

[0118] In 560, a second set of kernels can be mapped to a second set of crosspoint elements in the crossbar circuit. The second set of crosspoint elements can be connected to a second group of selection lines. Each of the second group of selection lines can be connected to a set of crosspoint elements that do not share a bit line or word line. As an example, as previously mentioned in relation to Figure 4B, the second set of kernels can be mapped to crosspoint elements connected to selection lines 431-2, 432-2, ..., and 433-2. As another example, as previously mentioned in relation to Figure 4C, the first set of kernels can be mapped to crosspoint elements connected to selection lines 435-1, 435-2, ..., and 435-N.

[0119] In 565, a first group of crosspoint elements can be enabled via the first group of selection lines. For example, an appropriate selection voltage can be applied to each of the first group of selection lines. As a more specific example, the crosspoint elements connected to selection lines 431-1, 432-1, ..., 433-1 in Figure 4B can be enabled. As another more specific example, the crosspoint elements connected to selection lines 434-1, 434-2, ..., 434-N can be enabled.

[0120] In 570, a first plurality of input signals can be provided to a first plurality of crosspoint elements. Each of the first plurality of input signals can represent a portion of the first input data that is convolved using the first plurality of kernels. The first plurality of input signals can be applied to the first plurality of crosspoint elements via a first plurality of word lines. Each of the first plurality of word lines can be connected to each of the crosspoint elements of the first plurality of crosspoint elements.

[0121] In 575, a first set of multiple output signals can be output through multiple bit lines of the crossbar circuit. The first output signals can represent the convolution result of the first input data and the first set of multiple kernels. These bit lines can be, for example, bit lines 413-1, 413-2, ..., 417-m, as previously described in relation to Figures 4B and 4C.

[0122] In 580, a second group of multiple crosspoint elements can be enabled via a second group of selection lines. For example, an appropriate selection voltage can be applied to each of the second group of selection lines. As a more specific example, the crosspoint elements connected to selection lines 431-2, 432-2, ..., 433-2 in Figure 4B can be enabled. As yet another more specific example, the crosspoint elements connected to selection lines 435-1, 435-2, ..., 435-N can be enabled.

[0123] In 585, a second set of input signals can be provided to a second set of crosspoint elements. The second set of input signals can represent second input data that is convolved using a second set of kernels. The second set of input signals can be applied to the second set of crosspoint elements via a second set of word lines. Each of the second set of word lines can be connected to each of the crosspoint elements of the second set of crosspoint elements.

[0124] In 590, a second set of output signals can be output through multiple bit lines of the crossbar circuit. The second output signals can represent the convolution result of the second input data and the second set of kernels.

[0125] For the sake of simplicity, the methods disclosed herein are illustrated and described as a series of actions. However, the actions according to this disclosure may occur in various orders, simultaneously or individually, and may occur together with other actions not presented or described herein. Furthermore, not all illustrated actions may be necessary to implement the methods according to the disclosed invention. In addition, those skilled in the art will understand and recognize that the methods can be alternatively represented as a series of interrelated states via state diagrams or events.

[0126] The terms “approximately,” “about,” and “substantially” can be used in some embodiments to mean within ±20% of the target specifications, within ±10% of the target specifications, within ±5% of the target specifications, and even within ±2% of the target specifications. The terms “approximately” and “about” may include target specifications.

[0127] The above description provides many details. However, it will become clear that disclosures can be made without these specific details. In some cases, well-known structures and devices are shown in block diagram form rather than in detail, in order to avoid obscuring the disclosure.

[0128] The terms "first," "second," "third," "fourth," etc., when used herein, mean labels that distinguish between different elements and do not necessarily have an orderly meaning according to their numerical designation.

[0129] The terms “example” or “exemplary” are used herein to mean serving as an example, case, or illustration. No aspect or design described herein as “example” or “exemplary” is necessarily construed as being preferable or advantageous to any other aspect or design. Rather, the use of the terms “example” or “exemplary” is intended to present a concept in a specific manner. When used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” In other words, unless otherwise stated in the specification or evident from the context, “X includes A or B” is intended to mean any natural inclusive sorting. That is, “X includes A or B” is satisfied under any of these cases: X includes A; X includes B; or X includes both A and B. In addition, the use of “a certain” and the absence of a quantity designation in this application and accompanying claims should generally be interpreted as meaning “one or more,” unless stated in the specification or the singular designation is clear from the context. Whenever “a certain implementation” or “one implementation” is referred to throughout this specification, it means that the specific features, structures, or characteristics described in relation to that implementation are included in at least one implementation. Therefore, it is not necessary that every instance of the phrase “a certain implementation” or “one implementation” appearing throughout this specification refers to the same implementation.

[0130] After reading the above description, it will be clear that many changes and modifications to this disclosure will be apparent to those skilled in the art; however, it will be understood that no particular embodiment illustrated and described by the illustrations is intended in any way to be considered limiting. Accordingly, references to details of various embodiments are not intended to limit the scope of the claims, and they merely enumerate the features that are considered to be part of the disclosure. [Explanation of Symbols]

[0131] 100 devices 101 First Crossbar Circuit 102 Second crossbar circuit, crossbar circuit 103 Word-Line (WL) Logic, WL Logic 104 Programming Circuit 105 Sensing Circuit 106 First Selection Logic 107 Second Selection Logic 111 Ward Line (WL), Ward Line 111-1, 111-2, 111-3, 111-4, 111-n, 111a, 111b, 111c Word Line 113, 113-1, 113-2, 113-3, 113a, 113b Bit Line (BL), Bit Line 120 Cross Point Elements 130, 140 selection lines 200 cross point elements 201 RRAM element 203 Transistors 211 bit line (BL), bit line 213 Selection Line (SEL), Selection Line 215 Ward Line (WL), Ward Line 300 Crossbar Circuits, Crossbar Arrays 400a, 400b, 400c, 400d Crossbar Circuit 401, 401-1, 401-2, 401-3, 401-m Word Line 403, 403-1, 403-2, 403-3, 403-m Word Line 405, 405-1, 405-2, 405-3, 405-m Word Line 407-1, 407-2, 407-3, 407-m bit line 411, 411-1, 411-2, 411-3, 411-m Input signals 413, 413-1, 413-2, 413-3, 413-m Input signals 415, 415-1, 415-2, 415-3, 415-m Input signals 417, 417-1, 417-2, 417-3, 417-m bit line 420, 421, 422, 423, 424, 425, 426 Cross point elements, cross bar arrays 430, 430-1, 430-2, 430-i Selection Line 430c Crossbar Circuit 431, 431-1, 431-2, 431-N Selection Line 432, 432-1, 432-2, 432-N selection lines 433, 433-1, 433-2, 433-N Selection Line 434, 434-1, 434-2, 434-N selection lines 435, 435-1, 435-2, 435-N selection lines 436 Selection Lines 450b Crossbar Array 460a, 460b Selection Logic 461 Part 1 461-1, 461-2, 461-3, 461-4, 461-5, 461-6, 461-N Selection Registers 462 Selection Register 463 Part 2 463-1, 463-2, 463-3, 463-4, 463-5, 463-6, 463-N Selection Registers 465 Part 3 465-1, 465-2, 465-3, 465-4, 465-5, 465-6, 465-N Selection Registers 610 images 610a Channel 1 610b Second channel 610c Third Channel 611 Part 1 611a Channel 1 611b Second channel 611c Third Channel 620, 621, 623, 625 kernels 630 Output Images 631 First element Outputs 640, 641, 643, 645 650 1x1x3 kernel Addresses 710, 711, 712, 713, 714, 715, 716, 717, 718, 719 Addresses 720, 721, 722, 723, 724, 725, 726, 727, 728, 729 Addresses 730, 731, 732, 733, 735, 736, 737, 738, 739, 743

Claims

1. A first crossbar circuit encompassing a first set of multiple crosspoint elements connecting multiple word lines and a first set of multiple bit lines, A second crossbar circuit that includes a second set of crosspoint elements connecting multiple word lines and a second set of multiple bit lines, A plurality of selection lines comprising a first selection line connecting a first group of the second plurality of crosspoint elements, a second selection line connecting a second group of the second plurality of crosspoint elements, and a third selection line connecting a third group of the first plurality of crosspoint elements, It includes, The first group of the second plurality of crosspoint elements is A first crosspoint element connecting a first bit line among the second plurality of bit lines and a first word line among the plurality of word lines, A second crosspoint element connecting the second bit line among the second plurality of bit lines and the second word line among the plurality of word lines, It includes, The second group of the second plurality of crosspoint elements is A third crosspoint element connecting the third word line and the first bit line, A fourth crosspoint element connecting the fourth word line and the second bit line, It includes, The third group of the first plurality of crosspoint elements is connected to the third bit line of the first plurality of bit lines and the plurality of word lines. Device.

2. The apparatus according to claim 1, further comprising a first selection logic connected to the first crossbar circuit and a second selection logic connected to the second crossbar circuit.

3. The apparatus according to claim 2, wherein the second selection logic is connected to the first group of the second plurality of crosspoint elements via the first selection line.

4. The apparatus according to claim 3, wherein the second selection logic is connected to the second group of the second plurality of crosspoint elements via the second selection line.

5. The aforementioned device further, Map the first plurality of elements of the plurality of depth convolution kernels to the first group of the second plurality of crosspoint elements, The second plurality of elements of the plurality of depth convolution kernels are mapped to the second group of the second plurality of crosspoint elements. Includes programming logic, The apparatus according to claim 4.

6. Furthermore, it includes word line logic connected to the aforementioned multiple word lines, The aforementioned word-line logic is, It receives input data to be convolved using multiple depth-direction convolution kernels, A plurality of input signals representing the aforementioned input data are generated, and, The plurality of input signals are applied to one or more crosspoint elements among the second plurality of crosspoint elements that are enabled by the second selection logic. The apparatus according to claim 2.

7. Furthermore, it includes sensing logic that generates a digital output based on a plurality of output signals output via the first plurality of bit lines and the second plurality of bit lines, The apparatus according to claim 6, wherein the digital output represents the convolution of the input data and the plurality of depth-direction convolution kernels.

8. The apparatus according to claim 1, wherein the first selection line and the second selection line are not parallel to the second plurality of bit lines.

9. The apparatus according to claim 8, wherein the first selection line and the second selection line are not parallel to the plurality of word lines.

10. At least one of the second plurality of crosspoint elements includes a transistor and a memristor. The apparatus according to claim 1.

11. The apparatus according to claim 1, wherein the first selection line and the second selection line are connected to the first selection register.

12. The apparatus according to claim 1, wherein the first selection line is connected to a first selection register, and the second selection line is connected to a second selection register.

13. By executing the program, the device will Mapping multiple kernels to multiple crosspoint elements of a crossbar circuit, The multiple crosspoint elements are enabled via multiple selection lines in the crossbar circuit, Applying multiple input signals to the multiple crosspoint elements, which represent input data to be convolved using the multiple kernels, Multiple output signals representing the convolution result between the input data and the kernel are output via multiple bit lines of the crossbar circuit. It includes, Mapping the plurality of kernels to the plurality of crosspoint elements of the crossbar circuit is By executing the aforementioned program, the device will The first element of the first kernel among the plurality of kernels is mapped to the first crosspoint element, The first element of the second kernel among the plurality of kernels is mapped to the second crosspoint element, It includes, The plurality of selection lines include a first selection line connecting a first group of the plurality of crosspoint elements and a second selection line connecting a second group of the plurality of crosspoint elements. The first group of the plurality of crosspoint elements includes a first crosspoint element connecting a first word line and a first bit line, and a second crosspoint element connecting a second word line and a second bit line. The second group of the plurality of crosspoint elements includes a third crosspoint element connecting the third word line and the first bit line, and a fourth crosspoint element connecting the fourth word line and the second bit line. method.

14. By executing the program, the device, Mapping multiple kernels to multiple crosspoint elements of a crossbar circuit, The multiple crosspoint elements are enabled via multiple selection lines in the crossbar circuit, Applying multiple input signals to the multiple crosspoint elements, which represent input data to be convolved using the multiple kernels, Multiple output signals representing the convolution result between the input data and the kernel are output via multiple bit lines of the crossbar circuit. It includes, Applying the plurality of input signals representing the input data that is convolved using the plurality of kernels to the plurality of crosspoint elements is By executing the aforementioned program, the device will Applying a first input signal to the first crosspoint element via the first word line of the crossbar circuit, The second input signal is applied to the second crosspoint element via the second word line of the crossbar circuit, It includes, The first input signal represents the first portion of the first channel of the input data, and the second input signal represents the second portion of the second channel of the input data. The plurality of selection lines include a first selection line connecting a first group of the plurality of crosspoint elements and a second selection line connecting a second group of the plurality of crosspoint elements. The first group of the plurality of crosspoint elements includes a first crosspoint element connecting a first word line and a first bit line, and a second crosspoint element connecting a second word line and a second bit line. The second group of the plurality of crosspoint elements includes a third crosspoint element connecting the third word line and the first bit line, and a fourth crosspoint element connecting the fourth word line and the second bit line. method.

15. Mapping the plurality of kernels to the plurality of crosspoint elements of the crossbar circuit further involves, By executing the aforementioned program, the device will Mapping the second element of the first kernel to the third crosspoint element, Mapping the second element of the second kernel to the fourth crosspoint element, The method according to claim 14, encompassing the present invention.

16. Enabling the multiple crosspoint elements via the multiple selection lines means that The method according to claim 13 or 14, wherein executing the program includes enabling the first group of the plurality of crosspoint elements by applying a first selection voltage to the first selection line.

17. Enabling the multiple crosspoint elements via the multiple selection lines means that The method according to claim 13 or 14, wherein the device enables the second group of the plurality of crosspoint elements by applying a second selection voltage to the second selection line by executing the program.

18. Applying the plurality of input signals representing the input data that is convolved using the plurality of kernels to the plurality of crosspoint elements is By executing the aforementioned program, the device will Applying a first input signal to the first crosspoint element via the first word line of the crossbar circuit, The second input signal is applied to the second crosspoint element via the second word line of the crossbar circuit, It includes, The first input signal represents the first portion of the first channel of the input data, and the second input signal represents the second portion of the second channel of the input data. The method according to claim 13.

19. By executing the aforementioned program, the device will Furthermore, the first and second data items stored within consecutive addresses are retrieved. It includes, The first input signal represents the first data item, and the second input signal represents the second data item. The method according to claim 14 or 18.

20. The apparatus according to claim 2, wherein the first selection logic is connected to the third group of the first plurality of crosspoint elements via the third selection line.

Images