Mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro

Abstract

The invention discloses a mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macros, which comprises a C-CIM supporting convolution operation, a plurality of weight reordering units (WRU), a main controller (MC), an adaptive weight copy unit (AWDU) and a global buffer area, and is characterized in that the global buffer area is connected with an external memory through a memory interface; the global buffer area is respectively connected with the weight reordering unit WRU, the self-adaptive weight replication unit AWDU and the storage macro C-CIM, the weight reordering unit WRU is respectively connected with the storage macro C-CIM and the main controller MC, the main controller MC is connected with the storage macro C-CIM, and the self-adaptive weight replication unit AWDU is connected with the storage macro C-CIM. The method aims at reducing redundant data movement and additional on-chip storage in the convolutional neural network operation process, improving the utilization rate of a multiply-accumulate arithmetic unit and improving the calculation efficiency of the lightweight convolutional neural network of the mobile terminal.

Description

Technical Field

[0001] The present invention relates to the technical field of hardware acceleration of convolutional neural networks, and in particular to a mobile lightweight convolutional neural network accelerator based on in-memory computing macros. Background Art

[0002] Lightweight CNN models provide a promising solution for CNN inference on mobile devices because of their reduced computational complexity and number of parameters, and have gained a lot of attention in academia and industry. However, the depthwise separable convolutions commonly used in lightweight CNN models have diverse convolution modes, which pose many challenges to traditional hardware architectures such as CPUs and GPUs. For example, in the well-known lightweight CNN model MobileNetV2, there are three different types of convolutions: standard convolution (STConv), depthwise convolution (DWConv), and pointwise convolution (PWConv). General-purpose computing units (e.g., CPU / GPU) cannot efficiently support all three types of convolutions at the same time. Previous studies focused on multi-type convolutions, which either showed suboptimal throughput and energy efficiency or incurred huge area costs. It is crucial to explore a new architecture to achieve high throughput and energy efficiency while reducing area costs, such as compute-in-memory (CIM) architectures.

[0003] Compute-in-memory architecture has become one of the research hotspots because it combines computing logic with storage to show great potential for high energy efficiency. Existing methods based on CIM macros can be mainly summarized into two categories: analog CIM macro cells and digital CIM macro cells. Although some works based on analog CIM macro cells support deep convolution and provide significant energy efficiency, their accuracy is easily affected by voltage and temperature. Digital CIM macro cells focus on specific operations, such as simultaneous multiplication and accumulation (MAC) and weight update based on 12-tube SRAM, supporting variable precision through bit string mechanism, and improving multiplier utilization through configurable addition trees. However, none of the existing digital CIM macro cells support direct convolution operations.

[0004] Mobile devices are participating in real-time CNN inference tasks such as image classification, face recognition, etc. However, traditional CNN models are computationally intensive and difficult to deploy easily on mobile devices due to resource constraints such as storage and power. Therefore, lightweight CNN models suitable for mobile devices have emerged, which decompose standard convolution into depthwise convolution and point-by-point convolution, thereby greatly reducing computational and memory overhead. Figure 1The figure shows three different types of convolutions in a standard convolutional lightweight convolutional neural network (CNN) model, where C is the number of input channels, N is the number of convolution kernels, K×K is the convolution kernel size, N is the number of output channels, R×Q is the output image size, and P is the number of feature maps. Figure 1 In the standard convolution (STConv) shown in (a), N The convolution kernel of dimension slides on the input feature map (Ifmap) to generate an output feature map (Ofmap) of size N×R × Q. This process can be transformed into and Matrix-matrix multiplication of size, such as Figure 1 As shown in (d) in , it has good data reusability. Figure 1 The depthwise convolution (DWConv) shown in (b) is P with a size of The convolution kernels slide on P Ifmaps respectively, and Ifmap cannot be reused between P convolution kernels. Its calculation is equivalent to matrix-vector multiplication (N = 1), and the cumulative length is short ( ).for Figure 1 The pointwise convolution (PWConv) shown in (c) has a kernel size of 1×1, so there is no data reuse in the convolution dimension and the accumulation length is also short ( ). Figure 1 (d) shows a schematic diagram of the conversion of convolution to general matrix multiplication. The three different convolution operations bring different computing characteristics and different data reuse patterns, which brings challenges to the deployment of lightweight CNN models on acceleration platforms for general matrix multiplication (GEMM). Figure 2 The paper shows the layer-by-layer multiplication-addition (MAC) utilization when processing MobileNetV2 using the most advanced digital memory macro units. There are three main challenges in processing lightweight CNN models on the traditional in-memory computing (CIM) architecture for general matrix multiplication: 1. When the algorithm converts standard convolution and depthwise convolution to GEMM (Conv2GEMM), a large amount of activation value movement will be generated due to redundant data movement and additional on-chip storage. 2. The MAC utilization of the multiply-accumulate operator of the depthwise convolution is extremely low (red dashed box), because the input feature map (Ifmap) is not reusable, resulting in a large number of subarrays in the column direction being idle when the weight fixed (WS) data flow is adopted, and the accumulation length is short (for example, 9), resulting in a large number of subarrays in the row direction being idle. 3. The MAC utilization of the multiply-accumulate operator of the pointwise convolution is low (orange dashed box), because the scale of most pointwise convolution layers is small, resulting in a large number of subarrays in the column / row direction being idle. Therefore, how to solve the challenges of processing lightweight CNN models on the traditional in-memory computing (CIM) architecture for general matrix multiplication has become a key technical problem that needs to be solved in the field of lightweight convolutional neural network acceleration on mobile terminals. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems of the prior art, a mobile lightweight convolutional neural network accelerator based on in-memory computing macros is provided. The present invention aims to reduce redundant data movement and additional on-chip storage during the convolutional neural network operation process, improve the utilization rate of the multiplication and accumulation operators, and improve the computing efficiency of the mobile lightweight convolutional neural network.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A mobile lightweight convolutional neural network accelerator based on in-memory computing macros includes a memory computing macro C-CIM supporting convolution operations, multiple weight reordering units WRU, a main controller MC, an adaptive weight copying unit AWDU and a global buffer, wherein the global buffer is connected to an external memory through a memory interface, the global buffer is respectively connected to the weight reordering unit WRU, the adaptive weight copying unit AWDU and the memory computing macro C-CIM, the weight reordering unit WRU is respectively connected to the memory computing macro C-CIM and the main controller MC, the main controller MC is connected to the memory computing macro C-CIM, and the adaptive weight copying unit AWDU is connected to the memory computing macro C-CIM.

[0007] Optionally, the storage and calculation macro C-CIM includes 64 sub-arrays of a read column address decoder, a write column address decoder, and a row address decoder. Each sub-array includes an SRAM array, a multiplier, and a configurable adder tree CAT. The read address signal RdA[3:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit read address signal RdA[1:0] is used as a column selection signal and is decoded by the read column address decoder to obtain a group of column selection signals C0~C3. The upper 2-bit read address signal RdA[3:2] is sent to the row address decoder to generate a row selection signal; the write address signal WrtA[7:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit write address signal WrtA [1:0] is used as a column selection signal and is decoded by the write column address decoder to obtain a group of column selection signals. The upper 6-bit write address signal WrtA[7:0] [7:2] is sent to the row address decoder to generate a row selection signal; the input of the row address decoder also includes 1024-bit write data Din[255:0][3:0] and a configurable write address interval parameter Winterval[3:0], and the write address interval parameter Winterval[3:0] is used to control the four rows of SRAM to be written to the SRAM at the same interval at the same time, and the corresponding operations include initializing the SRAM, updating the activation value and copying the weight; the multiplier is a 4b×1b multiplier, and the 4-way signal output by the SRAM array on the sub-array is respectively used as one input of the 4b×1b multiplier, and the other input of the 4b×1b multiplier comes from the 1024-bit input data Xin [255:0][3:0], and the input data Xin [255:0][3:0] Every 256 bits are broadcast to 16 sub-arrays; the output end of the multiplier is connected to a configurable adder tree CAT, the adder tree CAT has 6 stages, which is used for summing 64 products and can be configured as a GEMM mode with 1 channel of 64 inputs, 2 channels of 32 inputs, 4 channels of 16 inputs and a convolution mode with 7 channels of 9 inputs. The fourth, fifth and sixth stages of the adder tree CAT are provided with bit serial circular accumulators for accumulating partial sums or intermediate results.

[0008] Optionally, when the upper 6-bit write address signal WrtA[7:2] is sent to the row address decoder to generate a row selection signal, the row address decoder uses the upper 6-bit write address signal WrtA[7:2] as the first address addr0, and adds it to the write address interval parameter Winterval[3:0] as the address addr1, adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit as the address addr2, and adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit and left by 2 bits as the address addr3, thereby simultaneously generating 4 addresses addr0~addr3 with equal intervals as row selection signals.

[0009] Optionally, the bit serial circular accumulator uses a bit serial mechanism to sum the product terms and partial sums, including: receiving 3 input bit product terms p_sum, the partial sum Cin and the control signal ctrl, if the partial sum Cin exists, then setting the control signal ctrl to 2'11 at startup, so that the partial sum Cin is added to the 3 input bit product terms p_sum and stored in the circular shift register, and then setting the control signal ctrl to 2'01; if the partial sum Cin does not exist, then setting the control signal ctrl to 2'00 at startup, directly storing the input 3 input bit product terms p_sum in the circular shift register, and setting the control signal ctrl to 2'b01.

[0010] Optionally, the main controller MC adopts an adaptive weight copy strategy based on matrix size perception to copy the write data Din[255:0][3:0] as the weight matrix from the global buffer, including: automatically reorganizing the weight matrix according to the size of the weight matrix and selecting the required copy mode, the copy mode including no copy, row copy, column copy and row-column mixed copy, and after determining the copy mode, configuring the value of the write address interval parameter Winterval[3:0] according to the copy mode, and based on the corresponding copy mode, copying the write data Din[255:0][3:0] as the weight matrix from the global buffer and writing it into the SRAM of the storage and computing macro C-CIM.

[0011] Optionally, the main controller MC adopts a convolution mapping mechanism COMAS consisting of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage and computing macro C-CIM for convolution kernel to reduce on-chip data redundancy and reduce activation value communication.

[0012] Optionally, when the convolution kernel size is N×2×3×3, where N is the number of output channels, the size of the SRAM array in the storage macro C-CIM is 18×4, and the input feature map Ifmap mapping includes: initialization, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the first 9 rows of the SRAM4 column, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the SRAM The last 9 rows of the 4th column; activate the memory unit of the SRAM corresponding to the current sliding window and output it. When the activated SRAM corresponds to the current sliding window as the first sliding window, activate the first column of the SRAM; when the activated SRAM corresponds to the current sliding window as the second sliding window, activate the 4-9 and 13-18 rows of the first column of the SRAM and the 1-3 and 10-12 rows of the second column. When the activated SRAM corresponds to the current sliding window as the third sliding window, activate the 7-9 and 16-18 rows of the first column of the SRAM and the 1-6 and 10-15 rows of the second column; update the useless data consumed in the SRAM with the new data covered when the current sliding window slides down, including when the current sliding window slides down to the second sliding window, respectively using the input feature map Ifmap The data in the 4th row and 1st column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the first sliding window; when the current sliding window slides down to the third sliding window, the data in the 4th row and 2nd column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the second sliding window.

[0013] Optionally, when the convolution kernel size is N×2×3×3, where N is the number of output channels, the weight reordering includes: converting the convolution kernel of size N×2×3×3 into the form of a weight matrix, flattening the convolution kernel column by column and channel by channel, splicing them into vectors by channel, and then splicing them into a matrix W[0:17][0:1] by convolution kernel; outputting the corresponding weight matrix according to the output of SRAM in the storage macro C-CIM, when the convolution kernel slides within the row of the input feature map Ifmap, the weight matrix is ​​cyclically shifted within the convolution kernel with the convolution kernel column as the granularity; when the convolution kernel switches between rows of the input feature map Ifmap, the weight matrix is ​​cyclically shifted within the convolution kernel column with the element as the granularity.

[0014] Optionally, the main controller MC uses a convolution mapping mechanism COMAS consisting of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage and computing macro C-CIM to perform a convolution kernel to reduce on-chip data redundancy and reduce activation value communication, including calculating data redundancy according to the following formula: , , , In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap, and is the intermediate variable, To store the number of SRAM columns in the macro C-CIM, is the size of the convolution kernel, is the sliding step size of the convolution kernel, and also includes the activation value communication amount calculated according to the following formula : , In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap respectively.

[0015] Optionally, the baseline activation value communication volume is calculated according to the following formula when the im2col algorithm is used as the baseline : , In the above formula, and is the width and number of channels of the input feature map Ifmap, and are the convolution kernel size and sliding step size corresponding to the im2col algorithm respectively; the baseline activation value communication volume and activation value communication The comparison is used to determine the optimal quantification results of the convolutional mapping mechanism COMAS for data redundancy.

[0016] Compared with the prior art, the present invention mainly has the following advantages: the mobile lightweight convolutional neural network accelerator based on in-memory computing macros of the present invention includes a memory computing macro C-CIM supporting convolution operations, multiple weight reordering units WRU, a main controller MC, an adaptive weight copy unit AWDU and a global buffer, the global buffer is connected to the external memory through a memory interface, the global buffer is respectively connected to the weight reordering unit WRU, the adaptive weight copy unit AWDU and the memory computing macro C-CIM, the weight reordering unit WRU is respectively connected to the memory computing macro C-CIM and the main controller MC, the main controller MC is connected to the memory computing macro C-CIM, and the adaptive weight copy unit AWDU is connected to the memory computing macro C-CIM. Through the above structure, it can be used to reduce redundant data movement and additional on-chip storage in the convolutional neural network operation process, improve the utilization rate of the multiplication and accumulation operator, and improve the computing efficiency of the mobile lightweight convolutional neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of different types of convolutions in the prior art and conversion of convolutions into general matrix multiplications.

[0018] Figure 2 MAC utilization rate of the digital storage and computing macro unit in the prior art when processing MobileNetV2.

[0019] Figure 3 Schematic diagram of the accelerator structure in an embodiment of the present invention.

[0020] Figure 4 It is a schematic diagram of the structure of the storage and computing macro C-CIM in an embodiment of the present invention.

[0021] Figure 5 It is the original input feature map and convolution kernel mapped by the input feature map Ifmap in the embodiment of the present invention.

[0022] Figure 6 It is a schematic diagram of performing a convolution operation on the input feature map Ifmap on the C-CIM in an embodiment of the present invention.

[0023] Figure 7 Schematic diagram of the weight matrix after reordering during the convolution operation in an embodiment of the present invention.

[0024] Figure 8 This is a schematic diagram of block division when the input feature map Ifmap is too large in an embodiment of the present invention.

[0025] Fig. 9 Schematic diagram of eight forms of weight replication in an embodiment of the present invention.

[0026] Fig.10 Schematic diagram of deep convolution mapping in an embodiment of the present invention.

[0027] Fig.11 It is a flow chart of the convolution mapping mechanism COMAS in an embodiment of the present invention.

[0028] Fig.12 The figure is a flow chart of weight reordering in an embodiment of the present invention.

[0029] Fig.13 Schematic diagram of point-by-point convolution mapping in an embodiment of the present invention.

[0030] Fig.14 It is the layer-by-layer MAC utilization of MobileNetV2 in the embodiment of the present invention.

[0031] Fig.15 MAC utilization comparison diagram in the embodiment of the present invention.

[0032] Fig.16 This is a comparison chart of the activation value communication volume of the convolution mapping mechanism COMAS in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] like Figure 3 As shown, this embodiment provides a mobile lightweight convolutional neural network accelerator (CAMO for short) based on in-memory computing macros, including a memory computing macro C-CIM supporting convolution operations, multiple weight reordering units WRU, a main controller MC, an adaptive weight copy unit AWDU and a global buffer, the global buffer is connected to the external memory through a memory interface, the global buffer is respectively connected to the weight reordering unit WRU, the adaptive weight copy unit AWDU and the memory computing macro C-CIM, the weight reordering unit WRU is respectively connected to the memory computing macro C-CIM and the main controller MC, the main controller MC is connected to the memory computing macro C-CIM, and the adaptive weight copy unit AWDU is connected to the memory computing macro C-CIM.

[0034] In order to support both convolution and small-scale GEMM computing modes, such as Figure 4As shown, the storage and calculation macro C-CIM in this embodiment includes 64 sub-arrays of a read column address decoder, a write column address decoder, and a row address decoder. Each sub-array includes an SRAM array, a multiplier, and a configurable adder tree CAT. The read address signal RdA[3:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit read address signal RdA[1:0] is used as a column selection signal and is decoded by the read column address decoder to obtain a group of column selection signals C0~C3. The upper 2-bit read address signal RdA[3:2] is sent to the row address decoder to generate a row selection signal; the write address signal WrtA[7:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit write address signal WrtA [1:0] is used as a column selection signal and is decoded by the write column address decoder to obtain a group of column selection signals. The upper 6-bit write address signal WrtA[7:0] [7:2] is sent to the row address decoder to generate a row selection signal; the input of the row address decoder also includes 1024-bit write data Din[255:0][3:0] and a configurable write address interval parameter Winterval[3:0], and the write address interval parameter Winterval[3:0] is used to control the four rows of SRAM to be written to the SRAM at the same interval at the same time. The corresponding operations include initializing the SRAM, updating the activation value and copying the weight; the multiplier is a 4b×1b multiplier, and the 4-way signal output by the SRAM array on the sub-array is used as one input of the 4b×1b multiplier, and the other input of the 4b×1b multiplier comes from the 1024-bit input data Xin [255:0][3:0]. The input data Xin [255:0][3:0] broadcasts every 256 bits to 16 subarrays; the output of the multiplier is connected to a configurable adder tree CAT, which has 6 levels for summing 64 products and can be configured as GEMM mode with 1 channel 64 inputs, 2 channels 32 inputs, 4 channels 16 inputs, and convolution mode with 7 channels 9 inputs. The fourth, fifth, and sixth levels of the adder tree CAT are provided with bit serial circular accumulators for accumulating partial sums or intermediate results. In order to realize the flexible output of SRAM in C-CIM, the read address signal RdA is divided into two sections in this embodiment. The lower 2 bits are used as column selection signals to activate two adjacent columns of SRAM at the same time. After decoding, the upper 2 bits generate a set of row selection signals R0, R1, and R2, which select three rows of SRAM respectively. When Rn is in a high state, the first column is activated, and when it is in a low state, the second column is activated. For example, when C0, C1, C2, C3 = 0011, R0, R1, R2 = 001, the R2 row of the C2 column and the R0, R1 row of the C3 column in the output SRAM. Second, according to the previous discussion, the data reusability of Ifmap in DWConv is low, causing the input consumption rate to exceed its update rate, especially when the network becomes deeper and the sliding step size is greater than 1.After balancing the computing performance and hardware cost, this embodiment sets the bit width of the write data Din to 1024 bits and introduces a configurable write address interval parameter Winterval. After the write address is decoded, it can write four rows of SRAM at the same interval at the same time. For example, Winterval = 1, initialize SRAM; Winterval = 9, used to update the activation value and Winterval = 16 or 32 for weight replication. Then, the addition tree has 6 levels for 64 product summations, which can be configured as 1-way 64-input, 2-way 32-input, 4-way 16-input GEMM mode and 7-way 9-input convolution mode. The bit serial circular accumulators are distributed in the fourth, fifth and sixth levels of the addition tree for accumulating partial sums or intermediate results. Finally, the bit width of the input data Xin is set to 1024 bits, and broadcast to 16 sub-arrays every 256 bits. From the perspective of Xin, a 64×64 MAC array is divided into four 64×16 sub-arrays. The 4-way 16-input adder tree can be further decomposed into 16 sub-arrays of size 16 × 16 to better support small-scale GEMM.

[0035] like Figure 4 As shown, in this embodiment, when the high 6-bit write address signal WrtA[7:2] is sent to the row address decoder to generate a row selection signal, the row address decoder uses the high 6-bit write address signal WrtA[7:2] as the first address addr0, and adds it to the write address interval parameter Winterval[3:0] as the address addr1, adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit as the address addr2, and adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit and left by 2 bits as the address addr3, thereby simultaneously generating 4 addresses addr0~addr3 with equal intervals as row selection signals.

[0036] like Figure 4 As shown, the bit serial circular accumulator in this embodiment uses a bit serial mechanism to sum the product terms and partial sums, including: receiving 3 input bit product terms p_sum, partial sum Cin and control signal ctrl, if partial sum Cin exists, then setting the control signal ctrl to 2'11 at startup, so that the partial sum Cin is added to the 3 input bit product terms p_sum and stored in the circular shift register, and then setting the control signal ctrl to 2'01; if partial sum Cin does not exist, then setting the control signal ctrl to 2'00 at startup, directly storing the 3 input bit product terms p_sum in the circular shift register, and setting the control signal ctrl to 2'b01.

[0037] In order to improve the resource utilization of small-scale GEMM, a matrix size-aware adaptive weight copy strategy is designed. The main controller MC adopts an adaptive weight copy strategy based on matrix size awareness to copy the write data Din[255:0][3:0] as the weight matrix from the global buffer, including: automatically reorganizing the weight matrix according to the size of the weight matrix and selecting the required copy mode, the copy mode includes no copy, row copy, column copy and row-column mixed copy, and after determining the copy mode, configuring the value of the write address interval parameter Winterval[3:0] according to the copy mode, and based on the corresponding copy mode, copying the write data Din[255:0][3:0] as the weight matrix from the global buffer and writing it into the SRAM of the storage macro C-CIM. When loading weights from the global buffer, it automatically reorganizes the weight matrix according to the size of the weight matrix, such as Fig. 9 As shown. Depending on the size of the weight matrix, there are 8 cases, which can be further divided into 4 categories according to the different replication dimensions: (1) no replication (a); (2) row replication (b, c); (3) column replication (d); (4) row-column mixed replication (e, f, g, h). Then configure the write address interval parameter Winterval according to the number of rows replicated. For example, when no replication (a, d) is performed, the write address interval parameter Winterval = 1, when 2 rows are replicated (b, e, f), the write address interval parameter Winterval = 32, and when 4 rows are replicated (c, g, h), the write address interval parameter Winterval = 16. The replication process occurs when the weights are written from the global buffer to the SRAM of the C-CIM, so it does not increase the memory overhead.

[0038] In order to reduce on-chip data redundancy and reduce activation value communication, the main controller MC in this embodiment adopts a convolution mapping mechanism COMAS composed of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage macro C-CIM for convolution kernel to reduce on-chip data redundancy and reduce activation value communication. When the convolution kernel size is N×2×3×3, where N is the number of output channels, and the size of the SRAM array in the storage macro C-CIM is 18×4, the input feature map Ifmap mapping includes: initialization, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the first 9 rows of the SRAM4 column, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the SRAM The last 9 rows of the 4th column; activate the memory unit of the SRAM corresponding to the current sliding window and output it. When the activated SRAM corresponds to the current sliding window as the first sliding window, activate the first column of the SRAM; when the activated SRAM corresponds to the current sliding window as the second sliding window, activate the 4-9 and 13-18 rows of the first column of the SRAM and the 1-3 and 10-12 rows of the second column. When the activated SRAM corresponds to the current sliding window as the third sliding window, activate the 7-9 and 16-18 rows of the first column of the SRAM and the 1-6 and 10-15 rows of the second column; update the useless data consumed in the SRAM with the new data covered when the current sliding window slides down, including when the current sliding window slides down to the second sliding window, respectively using the input feature map Ifmap The data in the 4th row and 1st column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the first sliding window; when the current sliding window slides down to the third sliding window, the data in the 4th row and 2nd column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the second sliding window. When the convolution kernel size is N×2×3×3, where N is the number of output channels, the weight reordering includes: converting the convolution kernel of size N×2×3×3 into the form of a weight matrix, flattening the convolution kernel column by column and channel by channel, splicing them into vectors by channel, and then splicing them into a matrix W[0:17][0:1] by convolution kernel; outputting the corresponding weight matrix according to the output of SRAM in the storage macro C-CIM, when the convolution kernel slides in the row of the input feature map Ifmap, the weight matrix is ​​cyclically shifted in the convolution kernel with the convolution kernel column as the granularity; when the convolution kernel switches between rows of the input feature map Ifmap, the weight matrix is ​​cyclically shifted in the convolution kernel column with the element as the granularity.

[0039] The convolution mapping mechanism COMAS is very suitable for the calculation mode of the CIM macro and simplifies the hardware architecture design of this embodiment. To illustrate the convolution mapping mechanism COMAS, FIG. Figure 5 , Figure 6 , Figure 7 and Figure 8 An example is shown, including an Ifmap mapping on the SRAM array in the CIM macro, weight reordering of two convolution kernels with two-stage cyclic shift according to the pixel sequence output by the SRAM array, and input feature map Ifmap blocking when the width of the original input feature map Ifmap is too large. Figure 6 Shows Figure 5 The convolution process of the input feature map Ifmap on the SRAM array in (a) is shown in Figure 1. The size of the input feature map Ifmap is H × W × c = 5 × 12 × 2, the convolution kernel sliding step size S = 1, the SRAM size is 18×4, the row grouping is represented by R0 ~ 5, the column is represented by C0 ~ 3, and the horizontal axis represents Figure 5 The sliding window shown in the dashed box in (a) is shown in the figure. Fig.11 As shown, the calculation process can be divided into several key steps: First, initialize the SRAM with the first three rows of the input feature map Ifmap, and each group of three columns of pixels corresponds to a column of the SRAM. Secondly, activate the pixels corresponding to the current sliding window, for example, in the first sliding window, activate the C0 column and all row groups of the SRAM, and read all the pixels in the first column of the SRAM. In the second sliding window, the two columns C0 and C1 of the SRAM will be activated, but only the R1, R2 and R4, R5 row groups are activated, so the 3rd to 8th rows, 12th to 17th rows of the first column, and the 0th to 2nd rows, 9th to 11th rows of the second column are read, and the 0th and 9th rows of the first column of the SRAM are updated with the pixels of the 4th row of Ifmap. The whole process is performed iteratively, except that the sliding window switches between the rows of Ifmap, for example, in the 11th sliding window, the pixels are not reused in the column dimension, so the first row of pixels of the sliding window must be updated immediately. In addition, when the SRAM of the CIM macro cannot accommodate the original input feature map Ifmap (W>L), the input feature map Ifmap needs to be divided into multiple independent small blocks, which will generate a small amount of redundant data (overlapping areas between adjacent blocks), such as Figure 8 shown.

[0040] In this embodiment, the main controller MC uses a convolution mapping mechanism COMAS composed of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage macro C-CIM for convolution kernel to reduce on-chip data redundancy and reduce activation value communication, including calculating data redundancy according to the following formula: , , , In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap, and is the intermediate variable, To store the number of SRAM columns in the macro C-CIM, is the size of the convolution kernel, is the sliding step size of the convolution kernel, and also includes the activation value communication amount calculated according to the following formula : , In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap respectively.

[0041] This embodiment also includes calculating the baseline activation value communication volume according to the following formula when the im2col algorithm is used as the baseline : , In the above formula, and is the width and number of channels of the input feature map Ifmap, and are the convolution kernel size and sliding step size corresponding to the im2col algorithm respectively; the baseline activation value communication volume and activation value communication The comparison is used to determine the optimization quantification results of the convolution mapping mechanism COMAS on data redundancy. Taking the first layer of MobileNetV2 as an example, the baseline activation value of the input feature map Ifmap of 224×224×3 is is 3.36 M, while the activation value communication of the convolutional mapping mechanism COMAS The activation value communication volume is 1.63 M, which is 48.7% less than the baseline (im2col algorithm). And it has been verified that the activation communication volume of the convolution mapping mechanism COMAS on the entire MobileNetV2 is 76.78% lower than the baseline.

[0042] The weight reordering is intended to generate a weight vector corresponding to the pixel sequence of the SRAM array in the current sliding window. Figure 5 The two 3 × 3 × 2 convolution kernels in (b) are Figure 7 This is a schematic diagram of the weight matrix after reordering during convolution operation, such as Figure 7As shown, the columns of each channel are designated as a, b, and c respectively. Then, two situations are observed in this embodiment: (1) When the convolution kernel slides within the same row of the input feature map Ifmap, for example, in the 1st to 10th sliding windows, the weight vector is cyclically shifted within the convolution kernel channel with columns as the granularity. (2) When the convolution kernel switches between rows of the input feature map Ifmap, for example, in the 10th to 11th sliding windows, the weight vector is cyclically shifted within the convolution kernel column with elements as the granularity. Therefore, the weight reordering algorithm adopts a two-stage cyclic shift, and the shift step size is equal to the convolution window sliding step size S, as shown in Fig.12 As shown, the outer layer is based on the column granularity corresponding to the same row of the input feature map Ifmap, and the inner layer is based on the pixel granularity, which only occurs when switching between rows of the input feature map Ifmap.

[0043] The convolution kernel sizes of STConv and DWConv are both greater than 1, so they are both calculated in a convolutional manner. Fig.10 It illustrates how to map DWConv onto CAMO. For DWConv, the accumulation length is 9 and the addition tree is configured in 7-way 9-input mode to achieve parallelism within and between subarrays. Since the inputs between subarrays are not shared, it conforms to the feature of non-shared convolution kernels between Ifmaps in DWConv, so different input feature maps Ifmap are loaded within the subarray. Between subarrays, due to the input broadcast mechanism, different column blocks of the same Ifmap are loaded to achieve the reuse of convolution kernels within the input feature map Ifmap. Therefore, different output feature maps Ofmap will be generated in parallel within the subarray, and different column blocks of the same output feature map Ofmap will be generated in parallel between subarrays. Standard convolution follows a similar pattern, except that the accumulation length is 27 and the addition tree needs to be configured in 2-way 32-input mode.

[0044] Because the convolution kernel size is 1 × 1 and the convolution sliding windows do not overlap in PWConv, the GEMM mode is used for processing. There are two cases depending on the scale of PWConv. (1) The scale of PWConv is small, and the weight matrix needs to be copied, the image matrix needs to be split and reorganized accordingly, and the CAT needs to be configured according to the cumulative length. Fig.13As shown in (a), the projection layer of bottleneck1_1 of MobileNetV2 has an image matrix size of 12544 × 32 and a weight matrix of 32 × 16. The weight matrix needs to be copied 8 times and preloaded into the first column of the SRAM in C-CIM. The image matrix needs to be evenly divided into 8 blocks and reorganized into Xin, CAT configured as 2-way 32 inputs. (2) PWConv is large in scale. Due to the limitations of on-chip resources (such as the global buffer and C-CIM size), the image matrix and weight matrix need to be divided into blocks first and then processed according to (1). Fig.13 (b) in the figure is the projection layer of bottleneck4_3 of MobileNetV2, the image matrix size is 196×192, and the weight matrix size is 192×32. First, the weight matrix is ​​decomposed into three 64×32 sub-matrices, and then the three sub-weight matrices are copied twice and loaded into the first three columns of SRAM. At the same time, the image matrix is ​​reorganized, and the CAT is configured as 1-way 64 input.

[0045] This embodiment uses the configuration shown in Table 1 on the SOTA lightweight CNN model MobileNetV2 to evaluate CAMO, a mobile lightweight convolutional neural network accelerator based on in-memory computing macros.

[0046] Table 1: Configuration table

[0047] As shown in Table 1, this embodiment uses 64 sub-arrays and a 512 KB global buffer. The SRAM size of each sub-array is 64 × 4, each bit-cell is 4 bits, the addition tree is configurable, and the bit widths of Din and Xin are both 1024. This embodiment implements key modules such as WRU, AWDU, and MC in RTL, designs the layout of C-CIM, and synthesizes it in a 1GHz 12nm CMOS process. The power consumption is evaluated using the waveform file generated by the static timing analysis tool (STA) using zero-delay simulation. Table 2 shows the power consumption and area under the configuration of Table 1.

[0048] Table 2: Power consumption and area for the configuration in Table 1

[0049] Referring to Table 2, the chip area and power consumption of the memory computing macro C-CIM in this embodiment are only 16.64% and 13.4% of the mobile lightweight convolutional neural network accelerator CAMO based on the in-memory computing macro in this embodiment, respectively, and are dominated by the 512 KB global buffer, while the rest occupies only a small area and energy consumption.

[0050] Fig.14 The figure shows the layer-by-layer MAC utilization of MobileNetV2. Most layers have a utilization close to 100%, with only a few DWConv layers having a lower utilization of between 40% and 60%, because these layers have a sliding stride of 2 and very poor data reuse, making bandwidth a bottleneck. However, these layers account for a very small proportion of the entire model, so they have little impact on the overall performance.

[0051] In order to explain the high performance gain of the mobile lightweight convolutional neural network accelerator CAMO based on the in-memory computing macro of this embodiment, this embodiment compares the utilization rate of the mobile lightweight convolutional neural network accelerator CAMO based on the in-memory computing macro of this embodiment with three baselines, including: (1) Baseline-1, CAMO without CAT and COMAS, that is, the traditional CIM architecture. (2) Baseline-2, CAMO without CAT. (3) Baseline-3, CAMO without COMAS. Fig.15 As shown in the figure, the utilization of baseline-2 on STConv and DWConv is 2 times and 51.85 times higher than that of baseline-1, respectively, which is mainly due to the data reuse brought by COMAS. Due to CAT, the utilization of baseline-3 on PWConv is 2.23 times higher than that of baseline-1. The total utilization of CAMO, a lightweight convolutional neural network accelerator for mobile terminals based on in-memory computing macros in this embodiment, is 92.3%, which is 32.12 times higher than that of baseline-1. Fig.16 The activation value communication volume of STConv and DWConv in MobileNetV2 of CAMO, a lightweight convolutional neural network accelerator for mobile terminals based on in-memory computing macros, is compared with that of The baseline of the algorithm is compared. To better understand the advantages of COMAS, this embodiment is also Fig.16 Two hypothetical designs are added: CAMO_8 and CAMO_16, where the SRAM of C-CIM is set to 8 and 16 columns respectively. Due to the existence of COMAS, CAMO reduces the activation value communication by 76.78% compared with STConv and DWConv. In addition, as the number of SRAM columns in C-CIM increases, the activation value communication will be further reduced.

[0052] In summary, the mobile lightweight convolutional neural network accelerator based on in-memory computing macros in this embodiment implements a high-performance accelerator (CAMO) designed for lightweight CNN models based on a convolution capable-CIM (C-CIM) that supports convolution operations. CAMO consists of a convolution capable macro, multiple weight reordering units (WRUs), a master controller (MC), an adaptive weight duplication unit (AWDU), and a global buffer. Among them, C-CIM contains a flexible SRAM array and a configurable adder tree (CAT), supporting multiple computing modules. Each WRU is bound to a convolution input channel. MC generates read and write addresses for C-CIM and generates configurations for CAT. AWDU adaptively copies weights according to the size of the weight matrix and then writes them to C-CIM. The global buffer stores and reuses weights and output feature maps in convolution mode, as well as input feature maps and intermediate results in GEMM mode. CAMO supports different computing modes to adapt to various convolution types. Specifically, it proposes a direct convolution mode based on IS data flow for standard convolution and depthwise convolution, aiming to exploit data reuse and reduce activation value communication. In addition, it introduces a GEMM mode supporting weight replication and configurable addition trees for point-by-point convolution to improve MAC utilization. CAMO significantly improves MAC utilization and reduces activation value communication, thus achieving significant energy and area efficiency.

[0053] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A lightweight convolutional neural network accelerator for mobile terminals based on in-memory computing macros, characterized in that: It includes a storage-computing macro C-CIM supporting convolution operations, multiple weight reordering units WRU, a main controller MC, an adaptive weight duplication unit AWDU and a global buffer. The global buffer is connected to an external memory through a memory interface. The global buffer is respectively connected to the weight reordering unit WRU, the adaptive weight duplication unit AWDU and the storage-computing macro C-CIM. The weight reordering unit WRU is respectively connected to the storage-computing macro C-CIM and the main controller MC. The main controller MC is connected to the storage-computing macro C-CIM. The adaptive weight duplication unit AWDU is connected to the storage-computing macro C-CIM.

2. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 1, characterized in that: The storage and calculation macro C-CIM includes 64 sub-arrays of a read column address decoder, a write column address decoder, and a row address decoder. Each sub-array includes an SRAM array, a multiplier, and a configurable adder tree CAT. The read address signal RdA[3:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit read address signal RdA[1:0] is used as a column selection signal and is decoded by the read column address decoder to obtain a group of column selection signals C0~C3. The upper 2-bit read address signal RdA[3:2] is sent to the row address decoder to generate a row selection signal; the write address signal WrtA[7:0] of the storage and calculation macro C-CIM is divided into two sections. The lower 2-bit write address signal WrtA[1:0] is used as a column selection signal and is decoded by the write column address decoder to obtain a group of column selection signals. The upper 6-bit write address signal WrtA[7:0] [7:2] is sent to the row address decoder to generate a row selection signal; the input of the row address decoder also includes 1024-bit write data Din[255:0][3:0] and a configurable write address interval parameter Winterval[3:0], and the write address interval parameter Winterval[3:0] is used to control the four rows of SRAM to be written to the SRAM at the same interval at the same time, and the corresponding operations include initializing the SRAM, updating the activation value and copying the weight; the multiplier is a 4b×1b multiplier, and the 4-way signal output by the SRAM array on the sub-array is respectively used as one input of the 4b×1b multiplier, and the other input of the 4b×1b multiplier comes from the 1024-bit input data Xin [255:0][3:0], and the input data Xin [255:0][3:0] Every 256 bits are broadcast to 16 sub-arrays; the output end of the multiplier is connected to a configurable adder tree CAT, the adder tree CAT has 6 stages, which is used for summing 64 products and can be configured as a GEMM mode with 1 channel of 64 inputs, 2 channels of 32 inputs, 4 channels of 16 inputs and a convolution mode with 7 channels of 9 inputs. The fourth, fifth and sixth stages of the adder tree CAT are provided with bit serial circular accumulators for accumulating partial sums or intermediate results.

3. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 2, characterized in that: When the upper 6-bit write address signal WrtA[7:2] is sent to the row address decoder to generate a row selection signal, the row address decoder uses the upper 6-bit write address signal WrtA[7:2] as the first address addr0, and adds it to the write address interval parameter Winterval[3:0] as the address addr1, adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit as the address addr2, and adds it to the result of shifting the write address interval parameter Winterval[3:0] left by 1 bit and left by 2 bits as the address addr3, thereby simultaneously generating 4 addresses addr0~addr3 with equal intervals as row selection signals.

4. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 2, characterized in that: The bit serial circular accumulator adopts a bit serial mechanism to sum the product terms and partial sums, including: receiving 3 input bit product terms p_sum, partial sum Cin and control signal ctrl, if the partial sum Cin exists, then setting the control signal ctrl to 2'11 at startup, so that the partial sum Cin is added to the 3 input bit product terms p_sum and stored in the circular shift register, and then setting the control signal ctrl to 2'01; if the partial sum Cin does not exist, then setting the control signal ctrl to 2'00 at startup, directly storing the input 3 input bit product terms p_sum in the circular shift register, and setting the control signal ctrl to 2'b01.

5. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 2, characterized in that: The main controller MC adopts an adaptive weight copy strategy based on matrix size perception to copy the write data Din[255:0][3:0] as the weight matrix from the global buffer, including: automatically reorganizing the weight matrix according to the size of the weight matrix and selecting the required copy mode, the copy mode including no copy, row copy, column copy and row-column mixed copy, and after determining the copy mode, configuring the value of the write address interval parameter Winterval[3:0] according to the copy mode, and based on the corresponding copy mode, copying the write data Din[255:0][3:0] as the weight matrix from the global buffer and writing it into the SRAM of the storage and calculation macro C-CIM.

6. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 2, characterized in that: The main controller MC adopts a convolution mapping mechanism COMAS consisting of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage and computing macro C-CIM to perform convolution kernel to reduce on-chip data redundancy and reduce activation value communication.

7. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 6, characterized in that: When the convolution kernel size is N×2×3×3, where N is the number of output channels, the size of the SRAM array in the storage macro C-CIM is 18×4, and the input feature map Ifmap mapping includes: initialization, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the first 9 rows of the SRAM4 column, flattening the first three rows of the first channel of the input feature map Ifmap by column and writing them into the SRAM The last 9 rows of the 4th column; activate the memory unit of the SRAM corresponding to the current sliding window and output it. When the activated SRAM corresponds to the current sliding window as the first sliding window, activate the first column of the SRAM; when the activated SRAM corresponds to the current sliding window as the second sliding window, activate the 4-9 and 13-18 rows of the first column of the SRAM and the 1-3 and 10-12 rows of the second column. When the activated SRAM corresponds to the current sliding window as the third sliding window, activate the 7-9 and 16-18 rows of the first column of the SRAM and the 1-6 and 10-15 rows of the second column; update the useless data consumed in the SRAM with the new data covered when the current sliding window slides down, including when the current sliding window slides down to the second sliding window, respectively using the input feature map Ifmap The data in the 4th row and 1st column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the first sliding window; when the current sliding window slides down to the third sliding window, the data in the 4th row and 2nd column of the two channels of the input feature map Ifmap are used to update the 4th row and 13th row of data in the 1st column of the SRAM consumed by the second sliding window.

8. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 6, characterized in that: When the convolution kernel size is N×2×3×3, where N is the number of output channels, the weight reordering includes: converting the convolution kernel of size N×2×3×3 into the form of a weight matrix, flattening the convolution kernel column by column and channel by channel, splicing them into vectors by channel, and then splicing them into a matrix W[0:17][0:1] by convolution kernel; outputting the corresponding weight matrix according to the output of SRAM in the storage macro C-CIM, when the convolution kernel slides in the row of the input feature map Ifmap, the weight matrix is ​​cyclically shifted in the convolution kernel with the convolution kernel column as the granularity; when the convolution kernel switches between rows of the input feature map Ifmap, the weight matrix is ​​cyclically shifted in the convolution kernel column with the element as the granularity.

9. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 6, characterized in that: The main controller MC uses a convolution mapping mechanism COMAS composed of input feature map Ifmap mapping and weight reordering to input the input feature map Ifmap into the storage macro C-CIM to perform convolution kernel to reduce on-chip data redundancy and reduce activation value communication, including calculating data redundancy according to the following formula: ， ， ， In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap, and is the intermediate variable, To store the number of SRAM columns in the macro C-CIM, is the size of the convolution kernel, is the sliding step size of the convolution kernel, and also includes the activation value communication amount calculated according to the following formula : ， In the above formula, is the data redundancy, , and They are the width, height and number of channels of the input feature map Ifmap respectively.

10. The mobile terminal lightweight convolutional neural network accelerator based on in-memory computing macro according to claim 6, characterized in that: It also includes the calculation of the baseline activation value communication volume according to the following formula when the im2col algorithm is used as the baseline : ， In the above formula, and is the width and number of channels of the input feature map Ifmap, and are the convolution kernel size and sliding step size corresponding to the im2col algorithm respectively; the baseline activation value communication volume and activation value communication The comparison is used to determine the optimal quantification results of the convolutional mapping mechanism COMAS for data redundancy.

Images