Apparatus and method for matrix multiplication using in-memory processing

Abstract

The present invention discloses embodiments of an apparatus and method for matrix multiplication using in-memory processing (PIM). In an example, a device for matrix multiplication includes an array of blocks, each of which includes one or more PIM blocks. The PIM blocks may include mixed-mode PIM blocks configured as digital mode or analog mode. The PIM blocks configured as digital mode may perform operations associated with depthwise (DW) convolution. On the other hand, the PIM blocks configured as analog mode may perform operations associated with pointwise (PW) convolution. Depending on the calculation, a controller may be used to configure the PIM module to either digital mode or analog mode.

Description

Background Art

[0001] Embodiments of the present invention relate to processing-in-memory (PIM).

[0002] Ultra-low power machine learning processors are crucial for performing cognitive tasks in embedded systems, as power budgets are limited, for example in the case of batteries or energy harvesting sources. However, the data generated by deep convolutional neural networks (DCNNs) results in heavy traffic between memory and computing units in conventional von Neumann architectures and adversely affects the energy efficiency of these systems. As a promising solution to accelerate the execution of DCNNs, resistive random access memory (ReRAM) based on non-volatile PIM (NVPIM) has emerged. The high cell density of ReRAM allows large on-chip ReRAM arrays to be implemented on the chip to store the parameters of DCNNs, while suitable functions, such as vector-matrix multiplication (VMM), can be executed directly in the ReRAM array and its peripheral circuits. Summary of the invention

[0003] The following provides a simplified summary of one or more aspects to provide a basic understanding of these aspects. This summary is not an extensive overview of all intended aspects and is neither intended to identify key or critical elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that is presented later.

[0004] Embodiments of apparatus and methods for performing matrix multiplication using PIM are disclosed herein.

[0005] In one example, the apparatus may include a mixed-mode PIM device that can be configured in a digital mode or an analog mode. The mixed-mode PIM device may include: an input component configured to store a first vector and input data; a memory array configured as a peripheral logic operating in an analog mode and a digital mode; and a control circuit configured to enable the peripheral logic to perform a multiplication operation or an accumulation operation in the digital mode, and to perform a shift-add operation and generate a partial sum in the analog mode.

[0006] In another example, the PIM block may be configured to identify a group of memory units (MUs) based at least in part on a resolution associated with an analog-to-digital converter (ADC). In some aspects, the group of MUs may be located in a channel of a memory array, the channel comprising a group of columns of the memory array. In certain other aspects, each memory unit (MU) in the group of MUs consists of a group of adjacent memory units in a column in the group of columns. The PIM device may map each element in each matrix in a group of matrices to at least one MU in the group of MUs based at least in part on a precision of a weight associated with a first matrix. The PIM device may map the weights associated with each matrix in the group of matrices to the group of MUs as a vector. The PIM device may map each matrix in the group of matrices to a different column in a channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate embodiments of the present invention and, together with the description, further serve to explain the principles of the present invention and enable those skilled in the art to use the present invention.

[0008] Figure 1 A block diagram of a system-on-chip (SoC) device according to certain aspects of the present invention is illustrated.

[0009] Figure 2 Two exemplary computational flows that can be performed by a PIM block according to some embodiments of the present invention are illustrated.

[0010] Figure 3 An exemplary block-level computational flow for depthwise convolution according to some embodiments of the present invention is illustrated.

[0011] Figure 4A An exemplary kernel mapping flow according to certain aspects of the present invention is illustrated.

[0012] Figure 4B The invention illustrates an embodiment of the present invention that can be used Figure 4A An exemplary convolution process is performed using an exemplary kernel mapping process.

[0013] Figure 5A The data flow of a mixed-mode PIM block configured in analog mode according to certain aspects of the present invention is illustrated.

[0014] Figure 5B The data flow of a mixed-mode PIM block configured in digital mode according to certain aspects of the present invention is illustrated.

[0015] Figure 6 is a flow chart of an exemplary method for core mapping implemented by a PIM block configured in hybrid mode according to some embodiments of the present invention.

[0016] Embodiments of the present invention will be described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0017] Although the configuration and arrangement of the present invention have been discussed, it should be understood that this discussion is for illustration purposes only. It will be appreciated by those skilled in the art that other configurations and arrangements may be used without departing from the spirit and scope of the present invention. It will be apparent to those skilled in the art that the present invention may also be used for other multiple applications.

[0018] It should be noted that the "one embodiment", "an embodiment", "an example embodiment", "some embodiments", etc. mentioned in the specification of the present invention mean that the described embodiment may include specific features, structures or characteristics, but not every embodiment must include the specific features, structures or characteristics. In addition, such statements do not necessarily refer to the same embodiment. In addition, when specific features, structures or characteristics are described in conjunction with a certain implementation case, it is within the knowledge of those skilled in the art to implement such specific features, structures or characteristics in conjunction with other embodiments, regardless of whether it is explicitly stated herein.

[0019] In general, terms can be understood, at least in part, based on the context in which they are used. For example, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular, or to describe a combination of features, structures, or characteristics in the plural, based at least in part on the context. Similarly, terms such as "a," "an," and "the" can also be understood, at least in part, based on the context, to express singular usage or plural usage.

[0020] Deep convolutional neural networks (DCNNs) have been successfully applied to various fields, such as image classification. However, the success of DCNNs comes at the expense of high computational intensity, undesirable computational time, and the use of large data sets. Therefore, it may be challenging or even impractical to use currently available DCNN techniques in resource-constrained systems (e.g., mobile devices, edge computing nodes, smart embedded devices, etc.). In order to overcome computational and time limitations, hardware accelerators dedicated to DCNNs have been developed. One such hardware accelerator has been developed, including a non-volatile in-memory processing (NVPIM) architecture, which can reduce the time and computational resources used to perform DNCC calculations. For example, NVPIM can perform vector-matrix multiplication (VMM) operations through analog calculations in the memory array (e.g., a memory crossbar switch) itself, which can reduce computational time. In addition, NVPIM can reduce the overhead of data movement because the calculation occurs in situ inside the memory array. Lightweight DNCC models have also been developed to reduce the computational intensity and the number of model parameters. Depth-wise DCNN is such a lightweight model. In deep DCNN, traditional convolutional layers are replaced by a combination of depthwise (DW) convolutional layers and pointwise (PC) convolutional layers, which uses fewer multiplication and accumulation operations compared to other DNCC models.

[0021] Although the inference efficiency of DNCC operations can be improved using the NVPIM architecture and DW convolution, the overall efficiency decreases when mapping the DWDCNN to the existing NVPIM architecture. The existing NVPIM architecture is mainly designed for standard convolutions, where the PW convolution layer in the DWDCNN can be processed efficiently. Here, the PW convolution layer can be regarded as a standard convolution with a 1×1 kernel. However, due to the low utilization of the memory array (e.g., memory crossbar switch) when mapping the DW convolution layer to the NVPIM architecture as a VMM data layout, the DW convolution may not be performed efficiently when used in resource-constrained systems. One problem is that the feature map is only used for calculation in its own channel, which may result in a large number of unused memory cells in the memory array.

[0022] In order to achieve efficient reasoning for both DW and PW convolutional layers, a hybrid digital / analog mode NVPIM architecture is provided according to various embodiments of the present invention. For example, the present invention provides a digital mode NVPIM architecture that can process DW convolutional layers more efficiently than traditional techniques while using analog mode for calculations of PW convolutional layers. Additional details of the hybrid digital / analog mode NVPIM architecture are described below, for example, in conjunction with Figure 1-6 .

[0023] As an example but not limitation, the present invention takes ReRAM as an example to illustrate the design of the hybrid mode NVPIM architecture. However, a person of ordinary skill will understand that the following Figure 1-6 The described techniques of the present invention can be extended to any non-volatile memory (NVM) technology without departing from the scope of the present invention. For example, the described techniques of the present invention can also be applied to spin transfer torque magnetic random access memory (STT-MRAM), ferroelectric field effect transistor (FeFET) memory, to name just a few examples.

[0024] Figure 1 A block diagram of an embedded system-on-chip (SoC) device 100 according to certain aspects of the present invention is shown. The SoC device 100 may include, for example, a main memory 102, a main computer processing unit (CPU) 104, a system bus 106, an input-output (IO) processor 108, and an NVPIM accelerator 110. The NVPIM accelerator 110 may include a plurality of tiles 112 connected via a local bus 114. Each tile 112 may include one or more in-memory processing (PIM) tiles 116 configured to perform VMM calculations, an intra-tile buffer 118 configured to store intermediate data, and a functional unit (FU) 120 configured to process calculations other than VMM calculations.

[0025] Each PIM block 116 may include a memory array 124 composed of one or more memory crossbars 125a, 125b. Each memory crossbar 125a, 125b may include a plurality of word lines 128, a plurality of bit lines 126, and a memory cell 130 (e.g., a ReRAM memory cell, an STT-MRAM memory cell, a FeFET memory cell, etc.) located at each intersection of the word lines 128 and the bit lines 126. Each PIM block 116 may also include an input register 136 (e.g., an input component) configured to buffer input data, an output register 138 (e.g., an output component) configured to buffer output data, and a peripheral circuit such as a controller 134 configured to control and / or initiate operations performed at the PIM block 116. Additional peripheral circuits may also be included as local peripheral elements (PEs) 144, such as a multiplexer (MUX) 140, an analog-to-digital converter (ADC).

[0026] In some embodiments, the controller 134 can configure the PIM block 116 to operate in a hybrid mode when performing operations associated with DW convolution and to operate in an emulation mode when performing operations associated with PW convolution. In other words, the PIM block 116 can switch between the hybrid mode and the emulation mode depending on whether operations associated with DC convolution or PW convolution are performed. In the emulation mode, the local PE 144 can be configured to perform at least a portion of the operations associated with PW convolution, such as described below in conjunction with Figure 5A In digital mode, peripheral logic, such as local peripheral element (PE) 144, can be used to perform at least a portion of the operations associated with DW convolution, such as in conjunction with Figure 5B Just as described.

[0027] The logical value may be stored as the resistance of the memory cell 130, which may be switched between two or more levels by applying electrical stimuli having different amplitudes and durations. The memory cell 130 may be arranged with a structure of memory cross switches 125a, 125b that may be used to perform a VMM. The elements of the matrix (e.g., cores) may be represented as the conductance of the memory cell 130. The vector may be represented by a voltage applied to the input of the memory cross switches 125a, 125b. The voltage may be generated by a word line driver 132. Depending on the implementation, the voltage may be either binary or multi-level. When a voltage is applied to the word line 128, a current may be accumulated at the bit line 126. The bit line current may be collected at the MUX 140, which may be the output of the memory cross switches 125a, 125b. For example, the collected bit line current may represent a VMM. In Figure 1 In the example architecture shown, at least one ADC 142 may be used to convert a signal analogous to a bit line current into a digital value.

[0028] In some embodiments, the wordline driver 132 can be configured to activate all wordlines 128 in the memory cross switches 125a, 126c, and the number of ADCs 142 can be equal to the number of columns (e.g., bitlines 126) in the memory cross switches 125a, 125b. In this embodiment, the VMM calculation will be performed using a time complexity of O(1). However, activating all wordlines 128 concurrently may reduce the accuracy of the calculation. For example, high bitline 126 currents may cause IR drops. In addition, changes in the resistance of the memory cell 130 may cause cumulative errors. In order to reduce the inaccuracy of the VMM calculation, the wordline driver 132 can activate a subset of the wordlines 128 concurrently. In addition, because the ADC 142 may consume more power and substrate area than the memory cross switches 125a, 126b, multiple bitlines 126 can share one ADC 142 and only a limited number of bitline currents can be sampled in a single ADC cycle.

[0029] Figure 2 The figure shows an exemplary calculation flow that can be performed by the PIM block according to some embodiments of the present invention. The exemplary calculation flow 200 can be performed by any one or a combination of, for example, the PIM block 116, the memory array 124, the memory crossbar 125a, the memory crossbar 125b, the bit line 126, the word line 128, the memory cell 130, the word line driver (WD) 132, the controller 134, the input register 136, the output register 138, the MUX 140, the ADC 142, the local PE 144, the AND logic 502, the MUX circuit 504, and the shift adder 506.

[0030] In a deep DCNN, a standard convolution can be decomposed into a DW convolution and a 1×1 convolution, also referred to as a PW convolution 202. The DW convolution 204 can apply a filter to each input channel 206 for spatial accumulation. On the other hand, the PW convolution 202 can apply a 1×1 standard convolution to perform inter-channel feature map accumulation.

[0031] exist Figure 2 In the exemplary computational flow 200 of , the NN layer takes a tensor Tin of dimension [Ich×Hi×Wi] as input and produces a tensor Tout of dimension [Ich×Ho×Wo] as output feature map 208. Here, Hi / Ho, Wi / Wo, Ich, Och represent the spatial height, width, input and output depth, respectively. In the DW convolution 204, the kernel tensor Tk is of dimension [Ch×K×K], where K represents the kernel size. For example, the convolution of the depth layer can be described using equation (2):

[0032]

[0033] Figure 3 An exemplary block-level computational flow 300 for DW convolution according to some embodiments of the present invention is illustrated. Figure 3 The calculation process 300 in the above can be, for example, combined with Figure 1 The block 112 of the proposed NVPIM architecture described above is executed. Figure 3 , block 112 may include a PIM block 116 configured in a hybrid mode 302 and a PIM block 116 configured in an emulation mode 304. When configured in the hybrid mode 302, the PIM block 116 may be configured to perform DW convolution by performing multiplication and accumulation in a memory peripheral circuit (e.g., local PE 144). When configured in the emulation mode 304, the PIM block 116 may be configured to perform PW convolution by performing VMM calculations. Functional units (FUs) 122 may be configured to perform functions such as pooling, activation, feature map accumulation, and the like.

[0034] Referring to the computational flow 300, the feature map may be loaded (at 301) into the PIM block 116 configured in the hybrid mode 302. The PIM configured in the hybrid mode 302 may perform DW convolution based at least in part on the feature map to generate first output data. The first output data may be sent (at 302) to the FU122. The FU122 may generate an intermediate feature map by performing activation or pooling on the output data. The intermediate feature map may be sent (at 303) to the PIM block 116 configured in the simulation mode 304. The PM block 116 configured in the simulation mode 304 may perform PW convolution on the intermediate feature map to generate second output data. The second output data may be sent (at 304) to the FU122. The FU122 may perform activation or maximum pooling operations on the second output data to generate an output feature map 208, which is then sent (at 305) to a buffer 118 in the block 112 or in an adjacent block (not shown).

[0035] Figure 4A An exemplary DW core mapping process 400 according to certain aspects of the present invention is illustrated. The DW core mapping process 400 shown in FIG4 can be implemented by, for example, any one of the PIM block 116, the memory array 124, the memory crossbar 125a, the memory crossbar 125b, the bit line 126, the word line 128, the memory cell 130, the WD 132, the controller 134, the input register 136, the output register 138, the MUX 140, the ADC 142, the local PE 144, the AND logic 502, the MUX circuit 504, and the shift adder 506, or a combination thereof.

[0036] exist Figure 4A In the example shown, the weights of the DW convolutional layer can be mapped to a memory crossbar 125a. The configuration of the core 402 can be represented by [Ch, K, K], where Ch is the number of channels 404 in the memory crossbar 125a, and K is the core size. In this particular example, for simplicity, each memory cell 130 stores a binary value, such as "0" or "1". However, each memory cell 130 can store multiple levels of information without departing from the scope of the present invention. Similarly, in Figure 4A In the example shown, the resolution of ADC142 (RADC) is set to 4, but is not limited thereto. More generally, the resolution of ADC142 (RADC) may include any value less than or equal to the maximum logic value (e.g., number of bits) that can be sampled by ADC142 within the accuracy threshold during a single ADC cycle. Thus, depending on the maximum logic value sampled within the accuracy threshold, RADC may be less than or greater than 4 without departing from the scope of the present invention.

[0037] A group of memory units (MU) 408 located in the same channel 404 may be identified (at 401) based at least in part on RADC. For example, a group of RADC-adjacent memory units 130 located in the same column 406 (e.g., bit line 126) may be grouped into one MU 408. Again, the number of memory units 130 is grouped such that the logic value (e.g., 4 bits) of a single MU 408 may be accurately sampled by the ADC 142 during a single ADC cycle.

[0038] exist Figure 4A The cores 402 used in the illustrated example each include a plurality of elements (e.g., Figure 4A In the example of , there are nine MUs, and each element includes a weight (eg, an integer value). The mapping of elements to MUs may depend on the precision of the weights, as described below.

[0039] In some implementations, the precision of the weight may be equal to the RADC. Here, in each core 402, an element may be mapped (at 403) to a single MU 408. In some other implementations, the precision of the weight may be greater than the RADC. For example, the precision of the weight may be 8 bits and the RADC may be 4 bits. Here, in each core, an element may be divided into 4 least significant bits (LSBs) and 4 most significant bits (MSBs) and mapped (at 403') to two MUs 408 located in adjacent columns 406. In some other implementations, the precision of the weight may be less than the RADC. For example, the precision of the weight may be 2 bits and the RADC is 4 bits. Here, in each core, two elements at the same position of the two cores are grouped and then mapped (at 403") to one MU 408 in the memory array 124. Once the elements are mapped using one of the three example techniques described above, the DW core mapping process 400 may proceed as follows.

[0040] The weights in each core 402 may be mapped (at 405), for example, as a vector to K×K MUs 408 located in the same column 406 of the memory array 124. Each of the cores 402 may be mapped (at 407) to a single column 406 of the memory array 124. Thus, Figure 4A In the example shown, use the Ch column.

[0041] Using the above-described DW core mapping process 400, higher memory utilization can be achieved than when using conventional DW core mapping techniques. It should be understood that the DW core mapping process 400 is not limited to the case where the DW convolution layer fits into a single memory crossbar 125a of the memory array 124. Instead, the DW core mapping process 400 can be extended to the case where the DW convolution layer cannot fit into a single memory crossbar 125a of the memory array 124. Here, the DW convolution layer uses K×K×ChMU 408 to map all of its weights, which can be mapped to multiple memory crossbars 125a, 125b of the memory array 124.

[0042] Figure 4B is an exemplary DW convolution process 415 that may be performed by the PIM block 116 configured in digital mode according to some embodiments of the present invention. Figure 4B The DW convolution process 415 shown in FIG. 4 can be used, for example, in combination with Figure 4A The DW core mapping process 400 is described.

[0043] refer to Figure 4B , the weights (w) may be retrieved from the memory array 124 and calculated in the memory peripheral logic (e.g., the local PE 144). The weights (w) may be mapped to the memory array 124. The input feature map 410 may be maintained at the input register 136. The number of ADCs in each PIM block 116 may be defined as NADC. Thus, the MU (e.g., Figure 4A The weight (w) may be sent to the local PE 144. The corresponding feature map 410 may be fetched to the local PE 144. The local PE 144 may multiply the feature map 410 with the weight (w). The output result, such as the partial sum (Psum), may be sent to the output register 138.

[0044] As shown in cycles 0 and 1, the weights (w) may be read out using a row-major technique. In cycle N, the second row of MU may be read out and sent to the local PE 144 for multiplication. The Psum in the output register 138 may be fetched to the local PE 144 and added to the multiplication result. The local PE 144 may send the accumulated result to the output register 138.

[0045] Figure 5A5. The data flow 500 of the mixed-mode PIM block 116 configured in analog mode according to certain aspects of the present invention is illustrated. The data flow 500 can be associated with operations associated with PW convolution. In certain configurations, the controller 134 can configure the PIM block 116 in analog mode to perform PW convolution. The controller 134 initiates and / or controls one or more operations described below in conjunction with PW convolution.

[0046] In analog mode, a bit vector is sent from an input register 136 (e.g., an input component) (at 501) to WD 132. Using the bit vector, WD 132 can generate a binary voltage vector (e.g., Va for a logical value of "1"), which is applied to each word line 128. The bit line current accumulated on the bit line 126 can represent the partial sum sampled by ADC 142. Although not shown, when the input bit accuracy is 1, ADC 142 can send the partial sum value to the output register 138. Otherwise, the partial sum can be sent from ADC 142 (at 503) to the shift adder 506 for partial sum accumulation in the local PE 144. The partial sum accumulation can then be sent (at 505) to the output register 138 (e.g., an output component), where the partial sum accumulation can be stored.

[0047] Figure 5B The data flow 520 of the mixed-mode PIM block 116 configured in the digital mode according to certain aspects of the present invention is illustrated. The data flow 520 can be associated with operations associated with DW convolution. In certain configurations, the controller 134 can configure the PIM block 116 in the digital mode to perform the DW convolution. The controller 134 can initiate or perform one or more operations described below in conjunction with the DW convolution.

[0048] In digital mode, input data (e.g., input characteristic diagram) can be input (at 509) from input register 136 (e.g., input component) to local PE 144. WD 132 can generate a multi-level voltage vector (e.g., 1.0-8.0Va) to activate one MU 408. In this example, MU 408 includes four storage units 130. As described above in conjunction with Figure 4A As described above, the number of memory cells 130 grouped into MU 408 may be related to the resolution (RADC) of ADC 142. The bit line current accumulated at MU 408 may represent a logic value. Figure 5B The example shown shows that the bit line current represents a 4-bit logic value.

[0049] ADC 142 may sample the accumulated bit line current and sense a weight, which may be sent (at 511) to local PE 144. Local PE 144 may perform a multiplication operation using input data input from input register 136 and the weight received from ADC 142. Local PE 144 may generate an output feature map by performing an accumulation operation based at least in part on a result of the multiplication operation.

[0050] The local PE 144 of the present invention may include a bit-serial multiplier (e.g., 502 and 504) that performs a multiplication operation and a shift adder 506 that is configured to perform an accumulation operation using one or more results (e.g., outputs) from the bit-serial multiplier. The bit-serial multiplier included in the local PE 144 may include AND logic 502 and at least one multiplexer (MUX) circuit 504. The bit-serial multiplier may be added to the shift adder 506 that is also used in the analog mode. By reusing the shift adder 506 in the local PE for both digital and analog modes, the additional occupied space in the local PE 144 may be reduced. The output feature map accumulated by the shift adder 506 may be output (at 513) to the output register 138.

[0051] Figure 6 6 is a flow chart of an exemplary method 600 for DW core mapping implemented by a PIM block configured in a hybrid mode according to some embodiments of the present invention. For example, the operations described in conjunction with method 600 may be performed by, for example, any one or a combination of the PIM block 116, memory array 124, memory crossbar 125a, memory crossbar 125b, bit line 126, word line 128, memory cell 130, word line driver (WD) 132, controller 134, input register 136, output register 138, MUX 140, ADC 142, local PE 144, AND logic 502, MUX circuit 504, and shift adder 506 described in conjunction with the various exemplary implementations above. It can be understood that the operations shown in method 600 are not exhaustive, and other operations may also be performed before, after, or between the operations shown. In addition, some of the operations may be performed simultaneously or in parallel. Figure 6 The different orders are shown in . Figure 6 Some operations in may be optional, and these optional operations may be represented by dotted lines.

[0052] At 602, the PIM block may identify a group of MUs based at least in part on a resolution associated with the ADC. In some aspects, the group of MUs may be located in a channel of a memory subarray. In some other aspects, a channel may include a group of columns of the memory subarray. In some other aspects, each MU in the group of MUs is composed of a group of adjacent memory cells located in a column of the group of columns. For example, referring to Figure 4A , a group of MUs 408 (e.g., a group of MUs may include one or more MUs) located in the same channel 404 may be identified (at 401) based at least in part on the RADC. For example, a group of RADC-adjacent memory cells 130 located in the same column 406 (e.g., bit line 126) may be grouped into one MU 408. Furthermore, the number of memory cells 130 is grouped such that a logic value (e.g., 4 bits) of a single MU 408 may be accurately sampled by the ADC 142 during a single ADC cycle.

[0053] At 604, the PIM block may map each element of each matrix in the matrix set to at least one MU in the MU based at least in part on the precision of the weight associated with at least one matrix in the set of matrices (e.g., one or more cores). In some aspects, the mapping of the elements to the MUs may depend on whether the precision of the weight is equal to, greater than, or less than the number of identified MUs, as described below in conjunction with 606, 608, 610, 612. For example, when the precision of the weight is equal to the number of MUs, then the operation of 604 is performed using the operation of step 606. When the precision of the weight is greater than the number of MUs, then the operation of 604 is performed using the operation of step 608. When the precision of the weight is less than the number of MUs, then the operation of 604 is performed using the operations of steps 610 and 612.

[0054] At 606, when the precision of the weights is equal to the number of MUs (e.g., RADC), the PIM block may map each element in each matrix to a single MU in the MU group. For example, referring to FIG. 4, the precision of the weights may be equal to RADC. Here, in each core 402, one element may be mapped (at 403) to a single MU 408.

[0055] At 608, when the precision of the weights is equal to the number of MUs (e.g., RADC), the PIM block may map each element in each matrix to a pair of MUs in the set of MUs. In some aspects, the pair of MUs may be located in adjacent columns of the memory array. For example, referring to Figure 4A , in each core, an element may be divided into 4 LSB bits and 4 MSB bits and mapped (at 403 ′) to two MUs 408 located in adjacent columns 406 .

[0056] When the precision of the weights is greater than the number of MUs (eg, RADC), the operation moves to 610. At 610, the PIM block may group elements that occupy the same position in the two matrices. Figure 4A, the precision of the weights may be 2 bits, and the RADC is 4 bits. Here, in each core, two elements at the same position of two cores are grouped and then mapped (at 403") to one MU 408 in the memory array 124. Once the elements are mapped using one of the three example techniques described above, the DW core mapping process 400 can proceed as follows.

[0057] At 612, the PIM block may map the element grouping to a single MU in the MU group. Figure 4A , the precision of the weights may be 2 bits, and the RADC is 4 bits. Here, in each core, two elements at the same position of two cores are grouped and then mapped (at 403") to one MU 408 in the memory array 124. Once the elements are mapped using one of the three example techniques described above, the DW core mapping process 400 can proceed as follows.

[0058] At 614, the PIM block may map the weights associated with each matrix in the set of matrices as a vector to the set of MUs. Figure 4A , the weights in each core 402 may be mapped, for example as a vector (at 405 ) to K×K MUs 408 located in the same column 406 of the memory array 124 .

[0059] At 616, the PIM block may map each matrix in the set of matrices to a different column in the channel. Figure 4A , each of the cores 402 may be mapped (at 407) to a single column 406 of the memory array 124. Thus, at Figure 4A In the example shown, use the Ch column.

[0060] At 618, the PIM block may map a weight associated with at least one matrix in the set of matrices to one or more crossbars associated with the memory array. Figure 1 and 4A , the DW convolution layer uses K×K×ChMU408 to map all its weights, which can be mapped to multiple memory crossbar switches 125a, 125b of the memory array 124.

[0061] The foregoing detailed description of various specific embodiments is intended to fully disclose the general nature of the present invention so that others can easily modify / adjust these specific embodiments to adapt to various applications through basic common sense in the application field without excessive experimentation and without departing from the basic concepts of the present invention. Therefore, the above adjustments and modifications are based on the teachings and guidance of the present invention, and are intended to keep these modifications and adjustments within the meaning and scope of the equivalents of the embodiments described in the present invention. It can be understood that the words or terms used herein are for the purpose of description, so that people with professional knowledge can understand these words and terms under the enlightenment and guidance of the present invention, and should not be used to limit the content of the present invention.

[0062] The present invention uses functional modules to explain specific functions and specific relationships to achieve the description of the implementation cases in the present invention. For the convenience of description, the definition of the above functional modules is arbitrary. As long as the required specific functions and specific relationships can be achieved, other alternative definitions can also be adopted.

[0063] The Summary and Abstract sections may set forth one or more embodiments of the present invention but are not intended to include all exemplary embodiments contemplated by the inventor(s), and thus, are not intended to limit the scope of the present invention and the appended claims in any way.

[0064] The scope of the present invention should not be limited by any of the above-described embodiments, but should be defined in accordance with the following claims and their equivalents.

Claims

1. An in-memory PIM processing device, comprising: an input component configured to store at least a portion of the input data; A memory array including a plurality of memory units MU; Peripheral logic, configured to operate in either analog or digital mode; word line driver; as well as A control circuit configured to: In digital mode, it enables peripheral logic to perform multiplication or accumulation operations; as well as In analog mode, enable the peripheral logic to perform shift-add operations and generate partial sums, When in digital mode, the control circuit is further configured to: enabling a word line driver to generate a multi-level voltage vector to activate at least one memory unit MU in a memory array; wherein the bit line current is accumulated at the at least one memory unit MU after activation; as well as The bit line current is associated with a logic value which is stored as the resistance of the memory cell.

2. The process-in-memory (PIM) device of claim 1, wherein the at least one memory unit MU comprises a group of adjacent units in a column of the memory array.

3. The in-memory processing (PIM) device according to claim 1 further comprises an analog-to-digital converter (ADC), wherein: When in digital mode, the control circuit is further configured to: The analog-to-digital converter ADC is enabled to output a set of weights associated with the logic value to the peripheral logic.

4. The in-memory processing PIM device according to claim 3, wherein: When in digital mode, the control circuit is further configured to enable the peripheral logic to: Get input data from the input component; performing a multiplication operation based at least in part on a set of weights associated with the logic value and the input data; and An accumulation operation is performed based at least in part on one or more results of the multiplication operation to generate a feature map.

5. The in-memory processing (PIM) device of claim 1, wherein the digital mode supports operations associated with depthwise convolution.

6. The in-memory processing (PIM) device of claim 1, wherein the simulation mode supports operations associated with point-by-point convolution.

7. The processing-in-memory (PIM) device of claim 1, wherein the peripheral logic comprises a bit-serial multiplier configured to perform a multiplication operation.

8. The process-in-memory (PIM) device of claim 7, wherein the bit-serial multiplier comprises AND logic and MUX circuits.

9. The processing-in-memory (PIM) device of claim 7, wherein the peripheral logic further comprises a shift adder configured to perform an accumulation operation.

10. The in-memory processing PIM device according to claim 1, wherein: When in emulation mode, the memory array is configured to perform vector dot product operations.

11. A matrix multiplication method implemented by at least one in-memory processing (PIM) block configured in digital mode, the method comprising: identifying, by the controller at least in part based on a resolution associated with the analog-to-digital converter ADC, a group of memory units MU located in a channel of a memory array, the channel comprising a group of columns of the memory array, and each memory unit MU in the group of memory units MU comprising a group of adjacent memory units located in a column of the group of columns; mapping, by the controller, each element of each matrix of the set of matrices to at least one memory unit MU of the set of memory units MU based at least in part on a precision of a weight associated with at least one matrix of the set of matrices; mapping, by a controller, weights associated with each matrix of the set of matrices as a vector to the set of memory units MU; as well as Each matrix in the set of matrices is mapped by a controller to a different column in a channel.

12. The method of claim 11, wherein when the precision of the weight is equal to the number of memory units MU in the set of memory units MU, mapping each element of each matrix in the set of matrices further comprises: Each element in each matrix is ​​mapped to a single memory unit MU in the group of memory units MU.

13. The method of claim 11, wherein when the precision of the weight is greater than the number of memory units MU in the set of memory units MU, mapping each element in each matrix in the set of matrices further comprises: Each element in each matrix is ​​mapped to a pair of memory units MU in the group of memory units MU, the pair of memory units MU being located in adjacent columns of the memory array.

14. The method of claim 11, wherein when the precision of the weight is less than the number of memory units MU in the set of memory units MU, mapping each element in each matrix in the set of matrices further comprises: Group elements that occupy the same position in both matrices; as well as The element grouping is mapped to a single memory unit MU of the group of memory units MU.

15. An in-memory PIM processing device, comprising: A memory array including a plurality of memory units MU; as well as At least one controller coupled to the memory array and configured to: identifying a group of memory units MU in a memory array based at least in part on a resolution associated with an analog-to-digital converter ADC, the group of memory units MU being located in a channel of the memory array, the channel of the memory array comprising a group of columns of the memory array, and each memory unit MU in the group of memory units MU comprising a group of adjacent memory units located in a column of the group of columns; Mapping each element of each matrix of the set of matrices to at least one memory unit MU of the memory units MU based at least in part on the precision of the weights associated with at least one matrix of the set of matrices; mapping a weight associated with each matrix of the set of matrices as a vector to the set of memory units MU; as well as Each matrix in the set of matrices is mapped to a different column in the channel.

16. The in-memory processing (PIM) device of claim 15 , wherein when the precision of the weight is equal to the number of memory units MU in the group of memory units MU, the at least one controller is configured to map each element in each matrix in a group of matrices by: Each element in each matrix is ​​mapped to a single memory unit MU in the group of memory units MU.

17. The in-memory processing (PIM) device of claim 15 , wherein when the precision of the weight is greater than the number of memory units MU in the group of memory units MU, the at least one controller is configured to map each element in each matrix in a group of matrices by: Each element in each matrix is ​​mapped to a pair of memory units MU in the group of memory units MU, the pair of memory units MU being located in adjacent columns of the memory array.

18. The in-memory processing (PIM) device of claim 15, wherein when the precision of the weight is less than the number of memory units MU in the group of memory units MU, the at least one controller is configured to map each element in each matrix in a group of matrices by: Group elements that occupy the same position in both matrices; and The element grouping is mapped to a single memory unit MU of the group of memory units MU.

19. The in-memory processing (PIM) device of claim 15, wherein the at least one controller is further configured to: A weight associated with at least one matrix in the set of matrices is mapped to one or more crossbars associated with a memory array.

Images