In-memory computing macro devices and electronic devices

Abstract

This application discloses an in-memory computing macro device and an electronic device. The in-memory computing macro device includes an array of in-memory computing units comprising a plurality of in-memory computing units. First data is divided into at least two bit groups, the at least two bit groups including a first bit group and a second bit group, the first bit group being the most significant bit of the first data, the second bit group being the least significant bit of the first data, and the bit groups are respectively loaded into in-memory computing units in different columns of the in-memory computing unit array. The electronic device includes at least one in-memory computing macro and at least one processing circuit. The processing circuit is configured to receive parallel outputs corresponding to columns of the in-memory computing unit array and perform operations on the parallel outputs, wherein the parallel outputs include a plurality of corresponding groups, and each of the corresponding groups includes the most significant bit and the least significant bit of the output activation.

Description

TECHNICAL FIELD

[0001] The present application relates to a compute-in-memory (CIM) macro device and an electronic device. BACKGROUND

[0002] CIM is a technique to solve the bit-width and computation mismatch problem in the popular and widely used von-Neumann processor / architecture. The bit-width mismatch problem can be solved by combining memory and computation in the same block. However, the bit-width for computation is limited to the original width of the physical CIM macro, and thus the computation precision cannot be adjusted for higher precision application requirements. SUMMARY

[0003] The present application proposes a CIM macro device and an electronic device.

[0004] According to one of the example embodiments, the CIM macro device includes a CIM cell array having a plurality of CIM cells. A first data is divided into at least two bit groups, the at least two bit groups including a first bit group and a second bit group, the first bit group being a most significant bit of the first data, the second bit group being a least significant bit of the first data, and the at least two bit groups being loaded in CIM cells of different columns of the CIM cell array, respectively.

[0005] According to one of the example embodiments, the electronic device includes at least one CIM macro and at least one processing circuit. Each CIM macro includes a CIM cell array including a plurality of CIM cells. A first data is divided into at least two bit groups, the at least two bit groups including a first bit group and a second bit group, the first bit group being a plurality of most significant bits of the first data, the second bit group being a plurality of least significant bits of the first data. The at least two bit groups of the first data are loaded in CIM cells of different columns of the CIM cell array, respectively, and a second data is input to the CIM cells of the different columns of the CIM cell array after the first data is loaded in the CIM cells of the different columns. A processing circuit is configured to receive a plurality of parallel outputs corresponding to columns of the CIM cell array, respectively, and perform an operation on the plurality of parallel outputs, wherein the parallel outputs include a plurality of corresponding groups, and wherein each of the corresponding groups includes a plurality of most significant bits of an output activation and a plurality of least significant bits of the output activation.

[0006] However, it should be understood that this summary may not encompass all aspects and embodiments of this application, and therefore is not intended to limit or restrict in any way. Furthermore, this application will include improvements and modifications that will be obvious to those skilled in the art. Attached Figure Description

[0007] This document includes accompanying drawings to provide a further understanding of the present application, and the drawings are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present application and, together with this description, serve to explain the principles of the present application.

[0008] Figure 1 A schematic diagram of a traditional CIM macro device is shown.

[0009] Figure 2 A schematic diagram of an electronic device proposed in an exemplary embodiment according to this application is shown.

[0010] Figure 3 A schematic diagram of another electronic device proposed in one of the exemplary embodiments of this application is shown.

[0011] Figure 4 A schematic diagram of another electronic device proposed in one of the exemplary embodiments of this application is shown.

[0012] To make the above features and advantages of this application easier to understand, several embodiments are described in detail below with reference to the accompanying drawings. Detailed Implementation

[0013] Figure 1 A schematic diagram of a traditional CIM macro is shown.

[0014] Currently, neural network models can be used to compute more features from image input. The input feature map includes multiple input activations and is also called the input channel. Multiple weights applied to the input feature map are considered filters. By performing a convolution operation on the input channels using these filters, an output feature map is generated, which includes multiple output activations and is also called the output channel. (See reference...) Figure 1 As an example, a single macrodevice 110 would comprise a 256-row, 64-column CIM cell array, where the bit width for computation is a small, fixed number (e.g., 4), supporting only 4 bits of data storage in neural network computation (e.g., 4-bit (signed or unsigned) weights Wi(k, 3:0)). In this case, it is appropriate to preload the 4-bit weights Wi(k, 3:0) into the macrodevice 110. To apply the convolution operation to the signed or unsigned 8-bit input (also known as input activation) In(k), a 20-bit parallel output is produced. In the following embodiments, the bit width of the computation can be extended to a larger number (e.g. 8) without changing the physical CIM macro computation bit width, to facilitate computation of different bit width requirements.

[0015] To solve the problems highlighted, some embodiments of the present application will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all, embodiments of the application are shown. Indeed, the application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals are used throughout to refer to like elements.

[0016] Figure 2 A schematic diagram of an electronic device proposed in accordance with one of the example embodiments of the present application is shown.

[0017] Referring to Figure 2 The electronic device 200 includes at least one CIM macro 210 and at least one processing circuit 250. The CIM macro 210 includes a plurality of CIM cells formed as a CIM cell array of 256 rows and 64 columns (denoted by C201 to C264). The bit width of each CIM cell in the CIM cell array is 4 bits. A first data having a bit width greater than the bit width of each CIM cell (e.g. 8 bits) is divided into two bit groups, including a first bit group that is a plurality of most significant bits of the first data and a second bit group that is a plurality of least significant bits of the first data. The two bit groups of the first data are loaded in CIM cells located at different columns in the CIM cell array, respectively.

[0018] In one instance, after the first data is loaded in the CIM cells at different columns, a second data is input to the CIM cells at the aforementioned different columns of the CIM cell array, where the first data is a weight and the second data is an input activation. In another instance, after the first data is loaded in the CIM cells at different columns, a second data is input to the CIM cells at the aforementioned different columns of the CIM cell array, where the first data is an input activation and the second data is a weight.

[0019] In the present exemplary embodiment, a first bit group of the first data is loaded in one CIM cell of an odd column (e.g., one of columns C201, C203,... C263) of the CIM cell array, and a second bit group of the first data is loaded in one CIM cell of an even column (e.g., one of columns C202, C204,... C264) of the CIM cell array. As an example, an 8-bit unsigned weight can be divided into the first bit group and the second bit group, where the 4 most significant bits can be loaded in the odd column, and the 4 least significant bits can be loaded in the even column adjacent to the odd column. As another example, an 8-bit signed weight can be divided into the first bit group and the second bit group, where the 4 most significant bits can include a sign bit and can be loaded in the odd column, and the 4 least significant bits can be loaded in the even column adjacent to the odd column. For illustration purposes, W1M(k, 3:0) and W1L(k, 3:0) represent the 4 most significant bits (including the sign bit) and the 4 least significant bits of an 8-bit weight, respectively, where k = 1, 2,..., 256 e N. The rest can be inferred accordingly. As shown in FIG. 2, each row of the CIM cell array can store 32 8-bit weights (or 32 8-bit input activations). Figure 2

[0020] In the present exemplary embodiment, the processing circuit 250 is configured to receive a plurality of parallel outputs corresponding to columns C201 to C264 of the CIM cell array, respectively, and perform an operation on the plurality of parallel outputs. The parallel outputs include a plurality of corresponding groups, where each of the corresponding groups includes a plurality of most significant bits of an output activation and a plurality of least significant bits of the output activation. For example, the corresponding group for column C201 is the sum of W1M(1, 3:0) to W1M(256, 3:0), and the corresponding group for column C202 is the sum of W1L(1, 3:0) to W1L(256, 3:0). Each CIM cell of the CIM cell array is used to perform a convolution operation on a first data and a second data, where one of the first data and the second data is a weight, and the other is an input activation.

[0021] ​In this context, the parallel outputs include 32 corresponding groups, where each of the corresponding groups includes a plurality of most significant bits of an output activation and a plurality of least significant bits of the output activation. For illustration purpose, parallel outputs O1M and O1L representing a plurality of most significant bits and a plurality of least significant bits of an output activation O1, respectively, are considered as one corresponding group. For each of the corresponding groups, the processing circuit 250 shifts the plurality of most significant bits of the output activation and adds the shifted plurality of most significant bits of the output activation with the plurality of least significant bits of the output activation, where the output activation can be represented as Oi = OiM « 4 + OiL, i = 1, 2,..., 32 e N. In the case where the weights are signed, for each of the corresponding groups, the processing circuit 250 shifts the plurality of most significant bits of the output activation including the sign and adds the shifted plurality of most significant bits of the output activation with the plurality of least significant bits of the output activation.

[0022] In another exemplary embodiment, the bits of each of the weights can be divided into two or more bit groups. In this case, the number of bits of each of the bit groups can be determined based on the number of bits of each CIM cell, and each of the bit groups includes a portion of the bits of the first data and is loaded in different CIM cells belonging to different column groups of the CIM cell array, where the different column groups are grouped based on each two adjacent CIM cell columns in each column group being apart by an equal number of columns, that is, the spacing of the plurality of CIM cell columns in a column group is a common difference. As an example, for a 10-bit weight to be stored in a CIM cell array with a bit width of 4 bits, the 10-bit weight can be divided into three bit groups: a first bit group can include the most significant 4 bits, a second bit group can include the least significant 4 bits, and a middle group can include the remaining 2 bits. The number of bits of each bit group of the first data can not exceed the bit width of the CIM cell. As another example, for a 12-bit weight, the bits can be evenly divided into three bit groups: each of the first bit group, the middle bit group, and the second bit group can all include 4 bits. Note that in both examples, the first bit group can be loaded into the mthcolumn CIM cells among columns C201 to C264, the middle bit group can be loaded into the (m+1)thcolumn CIM cells among columns C201 to C264, and the second bit group can be loaded into the (m+2)thcolumn CIM cells among columns C201 to C264, where m e N. In this example, each two adjacent CIM cell columns in each column group are apart by three columns. The processing circuit 150 can receive a plurality of parallel outputs corresponding to columns of the CIM cell array, respectively, and perform operations on the plurality of parallel outputs in a similar manner as in the previous exemplary embodiments.

[0023] Figure 3FIG. 1 shows a schematic diagram of another electronic device according to one of the example embodiments of the present application.

[0024] Referring to Figure 3 The electronic device 300 includes at least one CIM macro 310 and at least one processing circuit 350. The CIM macro 310 includes a plurality of CIM cells forming a CIM cell array of 256 rows and 64 columns (denoted by C301-C364). The bit width of each CIM cell of the CIM cell array is 4 bits. In this example embodiment, each bit group includes a portion of bits of the first data, and each bit group is loaded in a CIM cell column group including consecutive columns. In this example embodiment, the first 32 consecutive CIM cell columns C301-C332 are referred to as a "first column group", and the last 32 consecutive CIM cell columns C333-C364 are referred to as a "second column group". The most significant bit and the least significant bit of each of the weights are loaded in different column groups of the CIM cell columns C301-C364, respectively. In this example embodiment, a plurality of most significant bits of each of the weights are loaded in the first column group C301-C332, and a plurality of least significant bits of each of the weights are loaded in the second column group C333-C364. As an example, an 8-bit unsigned weight can be divided into two columns, where the plurality of most significant bits can be loaded in a first column in the first column group C301-C332, and the plurality of least significant bits can be loaded in a first column in the second column group C333-C364. As another example, an 8-bit signed weight can include a sign bit, and can be divided into two columns, where the plurality of most significant bits can be loaded in a first column of the first column group C301-C332, and the plurality of least significant bits can be loaded in a first column of the second column group C333-C364. For illustration purposes, W1M(k,3:0) and W1L(k,3:0) denote 4 most significant bits and 4 least significant bits, respectively, of an 8-bit weight including a sign, where k = 1, 2,..., 256 e N. The rest can be inferred accordingly.

[0025] Similarly, the processing circuit 350 is configured to receive a plurality of parallel outputs corresponding to the CIM cell columns C301-C364 of the CIM cell array, respectively, and perform operations on the plurality of parallel outputs. The parallel outputs include a plurality of corresponding groups, where each of the corresponding groups includes a plurality of most significant bits of an output activation from the first column group C301-C332 and a plurality of least significant bits of the output activation from the second column group C333-C364. For illustration purposes, the parallel outputs O1M and O1L representing the plurality of most significant bits and the plurality of least significant bits of the output activation O1, respectively, are considered as one corresponding group. For each of the corresponding groups, the processing circuit 350 shifts the plurality of most significant bits of the output activation and adds the shifted plurality of most significant bits of the output activation to the plurality of least significant bits of the output activation, where the output activation can be represented as O1= O1M « 4 + O1L, i = 1, 2,..., 32 e N. In the case where the weights are signed, for each of the corresponding groups, the processing circuit 350 shifts the plurality of most significant bits of the output activation including a sign bit and adds the shifted plurality of most significant bits of the output activation to the plurality of least significant bits of the output activation.

[0026] Figure 4 A schematic diagram of another electronic device is shown, in accordance with one of the example embodiments of the present application.

[0027] Referring to Figure 4 The electronic device 400 includes a first CIM macro 410A, a second CIM macro 410B, and at least one processing circuit 450. The first CIM macro 410A includes a CIM cell array of a plurality of CIM cell columns C401A-C464A, and the second CIM macro 410B includes a CIM cell array of a plurality of CIM cell columns C401B-C464B. The bit width of each CIM cell of the first CIM macro 410A and the second CIM macro 410B is 4 bits. In another perspective, the first CIM macro 410A and the second CIM macro 410B form a CIM cell array, where a portion of the CIM cells of the CIM cell array belong to the first CIM macro 410A and another portion of the CIM cells of the CIM cell array belong to the second CIM macro 410B.

[0028] The most significant bits and the least significant bits of each of the weights are loaded in the first CIM macro 410A and the second CIM macro 410B, respectively. As an example, an 8-bit unsigned weight can be split into two columns, which are stored in the two CIM macros 410A and 410B, respectively, where the most significant bits can be loaded in the column of the first CIM macro 410A and the least significant bits can be loaded in the corresponding column of the second CIM macro 410B. As another example, an 8-bit signed weight can be split into two columns, which are stored in the two CIM macros 410A and 410B, respectively, where the most significant bits can include the sign bit and can be loaded in the column of the first CIM macro 410A and the least significant bits can be loaded in the corresponding column of the second CIM macro 410B. For illustration purposes, column C 401A of the first CIM macro 410A and column C 401B of the second CIM macro 410B are loaded by the most significant bits and the least significant bits of the same weight. The rest can be inferred accordingly.

[0029] The processing circuit 450 is configured to receive a plurality of parallel outputs corresponding to the first CIM macro 410A and the second CIM macro 410B, respectively, and perform an operation on the plurality of parallel outputs. The parallel outputs include 64 corresponding groups, where each of the corresponding groups includes the most significant bits of the output activations from the first CIM macro 410A and the least significant bits of the output activations from the second CIM macro 410B. For illustration purposes, the parallel outputs O1M and O1L representing the plurality of most significant bits and the plurality of least significant bits of the output activations O1, respectively, are considered as one corresponding group. For each of the corresponding groups, the processing circuit 450 shifts the plurality of most significant bits of the output activations and adds the shifted plurality of most significant bits of the output activations with the plurality of least significant bits of the output activations, where the output activations can be represented as Po = OiM « 4 + OiL, i = 1, 2,..., 64 ∈ N. In the case where the weights are signed, for each of the corresponding groups, the processing circuit 450 shifts the plurality of most significant bits of the signed output activations and adds the shifted plurality of most significant bits of the output activations with the plurality of least significant bits of the output activations.

[0030] In view of the above description, the proposed technology allows the expansion of the bit-width of the computation without changing the width of the physical macro computation in order to facilitate the implementation of computations with different bit-width requirements.

[0031] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the application. Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. Therefore, the scope of the application should be determined not with reference to the above description but should instead be determined with reference to the appended claims, along with their full scope of equivalents.

[0032] [CROSS-REFERENCE TO RELATED APPLICATIONS]

[0033] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 187,952, filed May 13, 2021. The entire disclosure of the above application is hereby incorporated by reference and made part of this specification.

Claims

1. An in-memory computing macro device, comprising: an array of in-memory computing units including a plurality of in-memory computing units, wherein a first data is divided into at least two bit groups, the at least two bit groups including a first bit group and a second bit group, the first bit group being a plurality of most significant bits of the first data, the second bit group being a plurality of least significant bits of the first data, the at least two bit groups of the first data being loaded in different columns of in-memory computing units of the array of in-memory computing units respectively, and a second data being input to the different columns of in-memory computing units of the array of in-memory computing units after the first data is loaded in the different columns of in-memory computing units, wherein the different columns of in-memory computing units of the array of in-memory computing units are used for convolution operation of the first data and the second data to generate a plurality of parallel outputs corresponding to the different columns.

2. The in-memory computing macro device of claim 1, wherein there is a second data input to the different columns of in-memory computing units of the array of in-memory computing units after the first data is loaded in the different columns of in-memory computing units, and wherein the first data is a weight and the second data is an input activation.

3. The in-memory computing macro device of claim 1, wherein there is a second data input to the different columns of in-memory computing units of the array of in-memory computing units after the first data is loaded in the different columns of in-memory computing units, and wherein the first data is an input activation and the second data is a weight.

4. The in-memory computing macro device of claim 1, wherein the first bit group of the first data is loaded in one in-memory computing unit of an odd column of the array of in-memory computing units, and the second bit group of the first data is loaded in one in-memory computing unit of an even column of the array of in-memory computing units.

5. The in-memory computing macro device of claim 1, wherein a number of bits of each of the at least two bit groups is determined based on a number of bits of each in-memory computing unit.

6. The in-memory computing macro device of claim 1, wherein each of the at least two bit groups includes a portion of the first data, and the at least two bit groups are loaded in different in-memory computing units belonging to different column groups of the array of in-memory computing units respectively, and wherein the different column groups are grouped based on each two adjacent columns of in-memory computing units of each column group being the same number of columns apart.

7. The in-memory computing macro device of claim 1, wherein each of the at least two bit groups includes a portion of the first data, and each bit group of the at least two bit groups is loaded in in-memory computing units of a column group including consecutive columns.

8. The in-memory compute macro device of claim 1, wherein a portion of the array of in-memory compute units belongs to a first in-memory compute macro and another portion of the array of in-memory compute units belongs to a second in-memory compute macro.

9. An electronic device comprising: at least one in-memory compute macro, wherein each of the at least one in-memory compute macro comprises an array of in-memory compute units, the array of in-memory compute units comprising a plurality of in-memory compute units, and wherein a first data is divided into at least two bit groups, the at least two bit groups comprising a first bit group and a second bit group, the first bit group being a plurality of most significant bits of the first data, the second bit group being a plurality of least significant bits of the first data, the at least two bit groups of the first data being loaded in in-memory compute units of different columns of the array of in-memory compute units, respectively, and a second data being input to the in-memory compute units of the different columns of the array of in-memory compute units after the first data is loaded in the in-memory compute units of the different columns, wherein the in-memory compute units of the different columns of the array of in-memory compute units are used for convolution operations of the first data and the second data to generate a plurality of parallel outputs corresponding to the different columns; and at least one processing circuitry configured to receive the plurality of parallel outputs corresponding to the columns of the array of in-memory compute units, respectively, and perform operations on the plurality of parallel outputs, wherein the parallel outputs comprise a plurality of corresponding groups, wherein each of the corresponding groups comprises a most significant bit of an output activation and a least significant bit of the output activation.

10. The electronic device of claim 9, wherein one of the first data and the second data is a weight and the other is an input activation.

11. The electronic device of claim 10, wherein a plurality of most significant bits and a plurality of least significant bits of each of the weights are loaded in odd columns and even columns of the columns of the in-memory compute units, alternately, and wherein for each of the corresponding groups, the at least one processing circuitry shifts the plurality of most significant bits of the output activation and adds the shifted plurality of most significant bits of the output activation with the plurality of least significant bits of the output activation.

12. The electronic device of claim 10, wherein the weights are unsigned numbers or signed numbers.

13. The electronic device of claim 9, wherein a number of bits of the first bit group and a number of bits of the second bit group are determined based on a number of bits of each in-memory compute unit.

14. The electronic device of claim 9, wherein each of the at least two bit groups includes a portion of bits of the first data, and the at least two bit groups are loaded in different memory-in-compute cells belonging to different column groups of the array of memory-in-compute cells, and wherein the different column groups are grouped based on each two adjacent columns of each column group being the same number of columns apart, and wherein for each of the corresponding groups, the at least one processing circuit shifts the plurality of most significant bits of the output activation and adds the shifted plurality of most significant bits of the output activation to the plurality of least significant bits of the output activation.

15. The electronic device of claim 9, wherein each of the at least two bit groups includes a portion of bits of the first data, and each bit group of the at least two bit groups is loaded in a memory-in-compute cell of a column group that includes consecutive columns, and wherein for each of the corresponding groups, the at least one processing circuit shifts the plurality of most significant bits of the output activation and adds the shifted plurality of most significant bits of the output activation to the plurality of least significant bits of the output activation.

16. The electronic device of claim 9, wherein a portion of the array of memory-in- compute cells belongs to a first memory-in-compute macro and another portion of the array of memory-in-compute cells belongs to a second memory-in-compute macro.

Images