Method of in-memory compute macro arrangement, computer readable medium, and electronic device
By dynamically adjusting the arrangement of CIM macros, the problem of unoptimized computational power and energy caused by fixed ratios in existing technologies is solved, achieving efficient convolution operation performance and optimized energy consumption.
Patent Information
- Application Number
- CN202210523375.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-05-13
- Filing Date
- 2022-05-13
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2042-05-13
AI Technical Summary
In existing technologies, the horizontal/vertical ratio of CIM macros is fixed and cannot be adjusted according to the number of input and output channels for different convolution calculations, resulting in unoptimized computational power and energy.
By obtaining the number and dimension information of CIM macros, as well as specifying the number of input and output channels of the neural network, the arrangement of CIM macros is dynamically adjusted to perform efficient convolution operations, taking into account latency, energy consumption, and utilization rate to optimize computational performance.
It enables dynamic adjustment of CIM macro arrangement according to the specific needs of convolutional layers, improving computational performance and energy efficiency, and reducing memory access energy costs.
Smart Images
Figure CN115345287B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a method for compute-in-memory (CIM) macro arrangement, a non-transitory computer readable medium, and an electronic device. BACKGROUND
[0002] CIM is a technology for solving the bandwidth and computation mismatch problem within the popular and widely used von-Neumann processor / architecture. The bandwidth mismatch problem can be solved by combining memory and computation in the same block. However, due to the high computation requirement of convolutional neural networks, multiple CIM macros will be used and should be used, but the horizontal / vertical ratio of multiple CIM macros is fixed and cannot be adjusted according to different numbers of input channels and output channels for different convolution computations, and the fixed ratio is not optimized for computation power / energy metrics. SUMMARY
[0003] The present application proposes a method for CIM macro arrangement, a non-transitory computer readable medium, and an electronic device.
[0004] According to one of the example embodiments, the method for CIM macro arrangement includes obtaining information of a number of CIM macros and information of a dimension of each of the CIM macros, obtaining information of a number of input channels and a number of output channels of a specified convolution layer of a specified neural network, and determining a CIM macro arrangement for arranging the CIM macros according to the number of the CIM macros, the dimension of each of the CIM macros, the number of the input channels and the number of the output channels of the specified convolution layer of the specified neural network, for convolution operation on the input channels to generate the output channels.
[0005] According to one of the example embodiments, an electronic device includes a CIM macro and a processing circuit. The CIM macro is arranged into a predetermined CIM macro arrangement based on a number of the CIM macros, a dimension of each of the CIM macros, and a number of input channels and a number of output channels of a specified convolution layer of a specified neural network. The processing circuit is configured to load weights in the arranged CIM macro, and input a plurality of input channels of one input feature map into the arranged CIM macro with the loaded weights to perform convolution operation for generating output activations of one of a plurality of output feature maps.
[0006] According to one of the example embodiments, the non-transitory computer readable medium stores a program that causes a computer to obtain information of a number of CIM macros and information of a dimension of each of the CIM macros; obtain information of a number of input channels and a number of output channels of a designated convolution layer of a designated neural network; and determine a CIM macro permutation for permuting the CIM macros for a convolution operation on the input channels to produce the output channels according to the number of the CIM macros, the dimension of each of the CIM macros, the number of the input channels, and the number of the output channels of the designated convolution layer of the designated neural network.
[0007] It is to be understood, however, that the summary of the application might not include all aspects and embodiments of the present application, and thus is not intended to limit or otherwise constrain the present application in any way. BRIEF DESCRIPTION OF DRAWINGS
[0008] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate the embodiments of the present application and, together with the description, serve to explain the principles of the present application.
[0009] Figure 1 A convolution in a convolutional neural network (CNN) is shown.
[0010] Figure 2 A flowchart of a proposed method for CIM macro permutation according to one of the example embodiments of the present application is shown.
[0011] Figure 3 Different CIM macro permutations according to one of the example embodiments of the present application are shown.
[0012] Figure 4 A convolution operation according to one of the example embodiments of the present application is shown.
[0013] Figure 5 A vertical CIM macro permutation according to one of the example embodiments of the present application is shown.
[0014] Figure 6 A horizontal CIM macro permutation according to one of the example embodiments of the present application is shown.
[0015] Figure 7 A block diagram of a system to perform a convolution operation according to one of the example embodiments of the present application is shown.
[0016] Figure 8A schematic diagram of a proposed electronic device is shown in accordance with one of the example embodiments in the present application.
[0017] So that the above features and advantages of the present application can be understood in detail, a few embodiments are described in detail with reference to the drawings. DETAILED DESCRIPTION
[0018] A common form of deep neural network (DNN) is a convolutional neural network (CNN) composed of multiple convolutional layers. In such a network, each convolutional layer takes input activations data and produces a higher level of abstraction of the input data, called a feature map, that retains necessary and unique information. Each of the convolutional activation layers in a CNN is primarily composed of a high-dimensional convolution. For example, Figure 1 A convolution in a CNN is shown.
[0019] Reference Figure 1 In this computation 100, the input activations of one layer are constructed as a set of two-dimensional (2D) input feature maps (IXxIYwith C channels) with multiple channels, each of which is called an input channel. Each input channel is convolved with a different 2D filter (i.e., kernel) from a 2D filter stack. This 2D filter stack is also called a single three-dimensional (3D) filter. Multiple 3D filters 110 (M filters with dimensions FXxFYxC) are convolved with the input feature maps 120. The convolution results for each point are summed across all input channels. The result of this computation is all output activations of one output channel, i.e., one output feature map with dimensions OXxOY. In other words, the 3D filters 110 are applied to the 3D input feature maps 120 (with batch size N) to create N output feature maps 130. For different layers in different DNNs, the number of input channels (denoted by C or IC) and the number of output channels (denoted by M or OC) will vary greatly depending on different parameter designs. However, the CIM macro arrangement for multiple CIM macros in the prior art is fixed and independent of the input / output channel numbers of the convolution.
[0020] To solve the above problems, some embodiments of the present application will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the present application are shown. Indeed, various embodiments of the present application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout.
[0021] Figure 2A flowchart showing a proposed method for CIM macro arrangement according to one of the example embodiments of the present application is shown. Figure 2 The steps in the above equation will be implemented by a computer system in the product development phase.
[0022] Referring to Figure 2 , information of a number of CIM macros and information of a dimension of each of the CIM macros are obtained (step S202), and information of a number of input channels and a number of output channels of a designated convolution layer of a designated neural network are obtained (step S204). Next, a CIM macro arrangement for arranging the CIM macros is determined for a convolution operation on the input channels to produce the output channels according to the number of the CIM macros, the dimension of each of the CIM macros, the number of input channels and the number of output channels of the designated convolution layer of the designated neural network (step S206). That is, by providing dynamic CIM shape configuration in hardware, a CIM shape decider implemented as a module in software can take the input channels and the output channels as input and decide a CIM shape configuration that can produce an optimal configuration.
[0023] In the present example embodiment, the CIM macro arrangement that can perform convolution of a plurality of filters with input channels is determined according to latency, energy consumption and utilization. The latency is associated with at least one of a latency of accessing a dynamic random access memory (DRAM), a latency for loading weights into the CIM macro, and a processing time of the CIM macro. Herein, the weights mean parameters of the filters, and the number of parameters of the filters is equal to FX x FY x IC x OC. The energy is a factor representing an energy cost for computing a convolution layer using a type of CIM macro arrangement, and the energy consumption is associated with an energy cost for accessing at least one memory including on-chip static random access memory (SRAM) in the same chip as the CIM macro and DRAM outside the chip. The utilization is a ratio of a used portion of the CIM macro to all CIM macros. For example, a ratio of DRAM:SRAM:CIM = 200:6:1 means that accessing the SRAM costs 6 times the energy cost of accessing the CIM on the basis of accessing the same amount of data.
[0024] In one case, the sum of all CIM macros provided by the determined CIM macro arrangement in the vertical dimension can enable the CIM macros of this arrangement to be adapted to be loaded into the input channels with the least number of batch loads to perform the convolution of the filters with the input channels of the specified convolution layer. In another case, the sum of all CIM macros provided by the determined CIM macro arrangement in the horizontal dimension can enable the CIM macros of this arrangement to be adapted to be loaded into the input channels with the least number of batch loads to perform the convolution of the filters with the input channels of the specified convolution layer.
[0025] To illustrate how multiple CIM macros can be effectively used to maximize computational performance, Figure 3 Different CIM macro arrangements are shown according to one of the example embodiments of the present application.
[0026] Referring to Figure 3 Each convolution layer of a CNN network can have a corresponding configuration of input / output channels. Assume that the number of CIM macros is 4 and each CIM macro has 256 rows and 64 columns of CIM cells. If the number of input channels is more than the number of output channels, using CIM macros in a vertical arrangement 310 can be more efficient. If the number of output channels is more than the number of input channels, using CIM macros in a horizontal arrangement 320 can be more efficient. If the number of input channels is equal to the number of output channels, using CIM macros in a square arrangement 330 can be more efficient.
[0027] For a better understanding, Figure 4 A convolution operation is shown according to one of the example embodiments of the present application.
[0028] Referring to Figure 4 Assume that for input data, IC = 512, OX = 28, and OY = 28, for filters F0 to F127, FX = 1, FY = 1, and OC = 128, IX = 1, IY = 1. In the case of using two macros, each having 256 rows and 64 columns, an efficient CIM macro arrangement for computing the convolution can be as shown in a vertical CIM macro arrangement according to one of the example embodiments of the present application as shown in FIG. 3B. Figure 5
[0029] Referring to Figure 5 64 filters F0, F1,..., F63 are pre-stored in the CIM macros. Input channels 1 to 256 are input to the first CIM macro (upper CIM macro) and input channels 257 to 512 are input to the second CIM macro (lower CIM macro). In other words, data of input cube 510 having dimension 1 x 1 x 512 (corresponding to Figure 4 The data of the input cube 410 in the figure is divided into two parts and input to each column (each column stores a filter) of the first and second CIM macros respectively for multiplication operation. For each of the first and second CIM macros, 256 multiplication results of each column are summed by the CIM macro as an output value, and an external circuit can add the two output values as the sum of 512 multiplication results to generate a convolution output. Thus, the first and second CIM macros as a whole can generate 64 convolution outputs. For example:
[0030] For the column of filter F0,
[0031] and for the column of filter F1,
[0032]
[0033] The convolution operation of the remaining 64 filters F65, F66,..., F127 will be similar.
[0034] In the same situation of using two CIM macros, each having 256 rows and 64 columns, the convolution can also be calculated using horizontally arranged CIM macros. In this situation, the first half number of input channels 1 to 256 can be input to each column of the 128 columns of the two horizontally arranged CIM macros (the 128 columns respectively pre-store 128 filters), and 256 multiplication results of each column are summed by the CIM macro as an output value. However, this output value cannot be a complete convolution output since the convolution operation of the second half number of input channels 257 to 512 has not been calculated. These output values (incomplete convolution outputs) must be stored in an accumulation buffer (SRAM or D flip-flop (DFF)). Until the convolution operation of the second half number of input channels 257 to 512 is also completed, the two parts of the incomplete convolution outputs are added to generate 128 convolution outputs. In this situation, more energy is spent on accessing the accumulation buffer, so it is less efficient than using two vertically arranged CIM macros.
[0035] Next, assume that the number of input channels is 128 and the number of output channels is 512. Since each macro has 256 rows (greater than 128), there is no need to arrange two CIM macros vertically. A single CIM macro will be able to complete the convolution operation of input channels 1 to 256 (i.e., the utilization rate of a single CIM macro is only 50%). In this situation, the efficient CIM macro arrangement for calculating convolution can be a horizontal CIM macro arrangement according to one of the exemplary embodiments of the present application as shown in FIG. 11. Figure 6
[0036] Referring toFigure 6 One-time loading of 128 filters F0 to F127 is performed to reduce the number of loading weights. Similarly, the data of the input cube 610 is input to each column of the first and second CIM macros, respectively, for performing the corresponding multiplication operations of the output.
[0037] Different products can apply different CNN architectures for data processing. For example, a surveillance system can apply CNN architecture A for data processing, while a surgical instrument can apply CNN architecture B for data processing. The appropriate CIM macro arrangement for a product can be predetermined by an offline tool based on the configuration of the convolution layers (i.e., Ox, Oy, IC, OC, FX, FY,...) of the CNN architecture selected by the product.
[0038] Once the CIM macro arrangement for a product is determined offline, Figure 7 A block diagram of a system performing convolution operations is shown in accordance with one of the example embodiments of the present application.
[0039] Referring to Figure 7 The CIM macros CIM0, CIM1, CIM2, and CIM3 are arranged in a predetermined CIM macro arrangement in the system 700. The weights (i.e., parameters of the filters) and instructions can be stored in the DRAM 720. When the system 700 receives an input feature map, the central processing unit (CPU) 710 can trigger the CNN scheduler 770 and the CIM macros CIM0 to CIM3 to perform convolution on a specified convolution layer of a specified neural network by hardware (e.g., DRAM 720, DDR controller 730, system on chip (SoC) bus 740, and data linear address (DLA) processor 75). The CNN scheduler 770 can receive the weights and input feature map (IFM) from the on-chip SRAM 760 and the CIM configuration from the instruction parser 750, load the weights and input feature map into CIM0 to CIM4 for convolution operations, receive the CIM outputs from CIM0 to CIM3, perform summation on all the CIM outputs to produce an intermediate result as an output feature map (OFM) of the current convolution layer to the SRAM 760, and wait for another new trigger.
[0040] In practical applications, Figure 8 A schematic diagram of a proposed electronic device is shown in accordance with one of the example embodiments of the present application, where the electronic device is assumed to be used by an end user.
[0041] Referring to Figure 8 The electronic device 800 includes a plurality of CIM macros 810 and a processing circuit 820. The CIM macros 810 are arranged in a predetermined CIM macro arrangement based on a number of the CIM macros, a dimension of each of the CIM macros, and a number of input channels and a number of output channels of a designated convolutional layer of a designated neural network. In the present exemplary embodiment, the predetermined CIM macro arrangement can be determined based on the method for CIM macro arrangement presented in Figure 2 The processing circuit 820 is configured to load weights in the arranged CIM macros and input a plurality of input channels of one input feature map into the arranged CIM macros with the loaded weights to perform a convolution operation for generating an output activation of one of output feature maps.
[0042] In an example, the weights of a filter can be first loaded into the CIM macros, and then the input channels (input feature maps) can be input into the CIM macros to perform the convolution operation. In another example, the inputs can be first loaded into the CIM macros, and then the weights can be input into the CIM macros to perform the convolution operation.
[0043] In the present exemplary embodiment, the processing circuit 820 loads weights of a plurality of filters in the arranged CIM macros based on the predetermined CIM macro arrangement, a number of the filters, a height and a width of each kernel of each of the filters, and a number of kernels in each filter, wherein each of the kernels of each filter is applied to a corresponding one of input channels of a designated convolutional layer of a designated neural network, respectively.
[0044] In one exemplary embodiment, the processing circuit 820 loads each of the filters into the arranged CIM macros column by column. The processing circuit 820 can determine whether to load the weights of the filters in the arranged CIM macros in batches based on a sum of the height and the width of each kernel and a horizontal dimension of the arranged CIM macros.
[0045] The present application also provides a non-transitory computer-readable recording medium that records a computer program to be loaded into a computer system to perform the steps of the proposed method. The computer program is composed of a plurality of program instructions. Once the program section is loaded into the computer system and executed by the computer system, the steps of the proposed method are completed.
[0046] In view of the foregoing description, the proposed technology enables efficient use of a plurality of CIM macros with an optimal configuration to maximize computational performance.
[0047] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the application. Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. Therefore, the scope of the application is intended to be limited only by the claims appended hereto and their equivalents.
[0048] [CROSS-REFERENCE TO RELATED APPLICATIONS]
[0049] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 187,952, filed May 13, 2021. The entire disclosure of the above application is incorporated herein by reference and forms a part of the specification.
Claims
1. A method for calculating macro arrangements in memory, comprising: Obtain information on the number of computational macros in multiple memories and information on the dimension of each of the multiple computational macros in multiple memories; To obtain information about the number of multiple input channels and the number of multiple output channels of a specified convolutional layer in a specified neural network; as well as An in-memory computation macro arrangement is determined based on the number of the plurality of in-memory computation macros, the dimension of each of the plurality of in-memory computation macros, the number of the plurality of input channels of the specified convolutional layer of the specified neural network, and the number of the plurality of output channels, for performing convolution operations on the plurality of input channels to generate the plurality of output channels.
2. The method of claim 1, wherein the step of determining the in-memory computational macro arrangement based on the number of the plurality of in-memory computational macros, the dimension of each of the plurality of in-memory computational macros, and the number of the plurality of input channels and the plurality of output channels of the specified convolutional layer of the specified neural network comprises: The in-memory computational macro arrangement is determined based on latency, energy consumption, and utilization rate, enabling the convolution of multiple filters with the multiple input channels.
3. The method of claim 2, wherein the determined in-memory computational macro arrangement provides the sum of the vertical dimensions of the plurality of in-memory computational macros, the plurality of in-memory computational macros being adapted to perform the convolution of the plurality of filters with the plurality of input channels of the designated convolutional layer with a minimum number of operations for batch loading of the plurality of input channels.
4. The method of claim 2, wherein the determined in-memory computational macro arrangement provides the sum of the horizontal dimensions of the plurality of in-memory computational macros, the plurality of in-memory computational macros being adapted to perform the convolution of the plurality of filters with the plurality of input channels of the specified convolutional layer with a minimum number of times for batch loading of the plurality of filters.
5. The method according to claim 2, The latency mentioned therein is associated with at least one of the following: dynamic random access memory latency, latency for loading weights into the plurality of in-memory computation macros, and processing time of the plurality of in-memory computation macros. The energy consumption is associated with the energy cost of accessing at least one memory, which includes on-chip static random access memory (SRAM) located on the same chip as the computational macros within the plurality of memories, and dynamic random access memory (DRAM) located off-chip. The utilization rate is the ratio of the used portion of the plurality of in-memory computed macros to all of the plurality of in-memory computed macros.
6. An electronic device comprising: Multiple in-memory computation macros, wherein the multiple in-memory computation macros are arranged into a predetermined in-memory computation macro arrangement based on the number of the multiple in-memory computation macros, the dimension of each of the multiple in-memory computation macros, and the number of multiple input channels and multiple output channels of a specified convolutional layer of a specified neural network. as well as The processing circuit is configured as follows: Weights are loaded into macros within the arranged plurality of memories; as well as Multiple input channels of an input feature map are fed into multiple in-memory computation macros with loaded weights to perform convolution operations for producing an output activation of one of multiple output feature maps.
7. The electronic device according to claim 6, The processing circuit loads the weights of the plurality of filters into the arranged plurality of in-memory computational macros based on the predetermined in-memory computational macro arrangement, the number of plurality of filters, the height and width of each of the plurality of kernels of each of the plurality of filters, and the number of the plurality of kernels of each of the plurality of filters, wherein each of the plurality of kernels of each filter is respectively applied to a corresponding one of the plurality of input channels of the specified convolutional layer of the specified neural network.
8. The electronic device according to claim 7, The processing circuitry therein loads each of the plurality of filters column by column into the arranged plurality of memory-based computation macros.
9. The electronic device according to claim 7, The processing circuitry determines whether to batch load the weights of the plurality of filters into the plurality of memory-computed macros based on the height and width of each core and the sum of the horizontal dimensions of the plurality of memory-computed macros arranged in order.
10. A non-transitory computer-readable medium storing a program that causes a computer to: Obtain information on the number of computational macros in multiple memories and information on the dimension of each of the multiple computational macros in multiple memories; To obtain information about the number of multiple input channels and the number of multiple output channels of a specified convolutional layer in a specified neural network; as well as An in-memory computation macro arrangement is determined based on the number of the plurality of in-memory computation macros, the dimension of each of the plurality of in-memory computation macros, the number of the plurality of input channels of the specified convolutional layer of the specified neural network, and the number of the plurality of output channels, for performing convolution operations on the plurality of input channels to generate the plurality of output channels.