Near-memory convolution operation circuit, convolutional neural network and chip system

Abstract

The invention provides a near-memory convolution operation circuit, a convolutional neural network and a chip system. The near storage convolution operation circuit is used for executing multiply-accumulate operation; the digital near storage convolution operation circuit comprises a logic control circuit, a memory array, a register group and a multiply-accumulate calculation unit, the multiply-accumulate calculation unit comprises a multiplier and an accumulator; the memory array comprises a magnetic random access memory sub-array and a static random access memory sub-array; the multiply-accumulate calculation unit is arranged on one side close to the memory array; and the register group is positioned between the multiply-accumulate calculation unit and the memory array.

Description

Technical Field

[0001] The present application relates to the field of circuit technology, and in particular to a near-memory convolution operation circuit, a convolutional neural network, and a chip system. Background Art

[0002] Since the birth of integrated circuits, their process technology has been following Moore's Law and has achieved continuous progress and development. However, as semiconductor processes are approaching physical limits, they have entered the post-Moore era, in which the further development of integrated circuit design no longer relies solely on the simple improvement of traditional process technology, but more on the emergence of various new technologies and their breakthroughs in application. In the post-Moore era, in order to break through the von Neumann energy efficiency bottleneck, related technologies have begun to actively explore new computing and storage architectures. In particular, the new architecture design methodology for computing in memory (CIM) has become a current research hotspot.

[0003] The development of in-memory computing has not only overturned the traditional computing architecture, but also greatly promoted the progress of processors, computing architectures, and computing models. It effectively solves the power consumption and speed bottlenecks caused by the separation of storage and computing in the von Neumann architecture, and realizes the design concept of integrated storage and computing. By shortening the distance between the computing unit and the memory, the in-memory computing method gives the memory computing power, thereby effectively solving the energy efficiency problem caused by frequent data movement in the traditional computing architecture, and further breaking through the bottleneck of energy efficiency and memory access speed. This design that integrates the computing process with the storage array blurs the boundary between the arithmetic unit and the storage array.

[0004] STT-MRAM (Spin Transfer Torque Magnetic Random Access Memory) is the most mature large-scale integration method for spin electronic devices in the current process, and it has shown significant advantages in non-volatile and high-density storage. However, due to the device characteristics of STT-MRAM itself, it is not suitable for direct use in multi-bit operations. At the same time, reading the accumulated results by magnetoresistive accumulation also requires complex sampling sensing technology. In contrast, SRAM (Static Random Access Memory) achieves storage through voltage, is naturally compatible with digital computing, and exhibits high performance and is easy to work with digital computing units.

[0005] Therefore, exploring the in-memory computing implementation of MRAM (Magnetic Random Access Memory)-SRAM hybrid storage has become a feasible computing architecture. This hybrid storage aims to leverage the non-volatility and high-density storage advantages of MRAM to achieve effective storage of weights; at the same time, it uses the high performance of SRAM and its natural compatibility with digital computing units to complete data caching and computing processes. Through this combination, it is expected to achieve a low-power, high-precision in-memory computing implementation method, providing new possibilities for future computing system architectures. Summary of the invention

[0006] In view of this, the present application provides a near-memory convolution operation circuit, a convolutional neural network and a chip system, which are used to provide an in-memory computing circuit with low standby power consumption and high computing accuracy.

[0007] In a first aspect, an embodiment of the present application provides a near-memory convolution operation circuit, which is used to perform multiplication and accumulation operations; the near-memory convolution operation circuit is used to perform multiplication and accumulation operations; the near-memory convolution operation circuit includes: a logic control circuit, a memory array, a register group and a multiplication and accumulation calculation unit; the multiplication and accumulation calculation unit includes: a multiplier and an accumulator; the memory array includes a magnetic random access memory subarray and a static random access memory subarray; the multiplication and accumulation calculation unit is arranged on a side close to the memory array; the register group is located between the multiplication and accumulation calculation unit and the memory array.

[0008] Optionally, the static random access memory loads weight data from the magnetic random access memory, and the static random access memory stores a plurality of different weight data simultaneously.

[0009] Optionally, the input data is directly input into the register group, and the register group stores one piece of weight data corresponding to the input data each time.

[0010] Optionally, the multiplication-accumulation calculation unit is implemented by a digital calculation circuit, which includes: an encoding circuit, a weight control circuit, a partial product generating circuit, a compression circuit and an adder circuit; the encoding circuit is used to encode the input data; the weight control circuit is used to control the weight data; the partial product generating circuit is used to generate a partial product result for the encoded input data and the controlled weight data; the compression circuit includes compressors with different ratios for compressing the partial product results; the adder circuit includes a carry adder and a shift accumulator for completing the multiplication-accumulation calculation.

[0011] Optionally, the weight data is stored in the magnetic random access memory in the form of a complement code.

[0012] Optionally, the encoding circuit performs Booth encoding on the input data.

[0013] Optionally, a first control signal controls data reading from the static random access memory and data writing to the magnetic random access memory; when the first control signal is at a low level, the weight data is written into the magnetic random access memory; when the first control signal is at a high level, the static random access memory reads the weight data from the magnetic random access memory; and a second control signal controls the digital computing circuit to enter a computing operation.

[0014] Optionally, the priority of the second control signal is higher than the priority of the first control signal.

[0015] In a second aspect, an embodiment of the present application provides a convolutional neural network, comprising a near-memory convolution operation circuit as described in any one of the first aspects, wherein the convolutional neural network is used to complete the multiplication and accumulation operations of the convolutional neural network according to the near-memory convolution operation circuit.

[0016] In a third aspect, an embodiment of the present application provides a chip system, comprising the near-memory convolution operation circuit as described in any one of the first aspects, and the chip system is used to complete the multiplication and accumulation operation of the chip system according to the near-memory convolution operation circuit.

[0017] The near-memory convolution operation circuit provided in the embodiment of the present application avoids the data transfer operation brought by the von Neumann architecture through near-memory, uses MRAM-SRAM for heterogeneous integrated weight storage, and uses a near-memory multiplier combined with a compressor-adder to implement a near-memory calculation method with low power consumption and high calculation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A It is a schematic diagram of a traditional digital computing architecture provided by an embodiment of the present application;

[0019] Figure 1B It is a schematic diagram of the digital domain CIM architecture provided by an embodiment of the present application;

[0020] Figure 2A It is the simulation domain CIM computing architecture provided by the embodiment of the present application;

[0021] Figure 2B The embodiment of the present application provides a schematic diagram of a simulation domain CIM architecture based on time calculation;

[0022] Figure 3 is a MRAM-SRAM digital computing circuit diagram provided in an embodiment of the present application;

[0023] Figure 3A is a schematic diagram of a 3:2 compressor provided in an embodiment of the present application;

[0024] Figure 3B is a schematic diagram of a 4:2 compressor provided in an embodiment of the present application;

[0025] Figure 3C is a detailed structural diagram of a digital computing circuit provided in an embodiment of the present application;

[0026] Figure 4 It is a schematic diagram of data writing, transfer and calculation waveforms provided in an embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of a near-memory convolution operation circuit of an MRAM-SRAM hybrid integrated integer quantized convolution operation in-memory computing unit provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the unit layout of the IQC-CIM provided in the embodiment of the present application;

[0029] Figure 7 Schematic diagram of the VGG convolutional neural network model provided in the embodiment of the present application;

[0030] Fig. 8A Schematic diagram of the relationship between power consumption per bit of data transfer and VDD provided in an embodiment of the present application;

[0031] Figure 8B This is a schematic diagram of the power consumption ratio of each part in the data transfer-calculation process provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the field of in-memory computing, in-memory computing in the digital domain has higher computing accuracy, greater flexibility, and higher computing parallelism than in the analog domain, and can fully utilize the advantages brought by technological development. The implementation of CIM in the digital domain is mainly based on digital circuit calculations, while CIM in the analog domain is based on analog calculations, see Figure 1A and Figure 1B . Figure 1A is a schematic diagram of a traditional digital computing architecture provided in an embodiment of the present application, Figure 1BIt is a schematic diagram of the digital domain CIM architecture provided by an embodiment of the present application. As can be seen from the traditional digital computing architecture shown in Figure 1, the weights are stored in the storage module, and the MAC unit (Multiply-Accumulate) loads the weights related to the current calculation step from the storage module. For example, W0 and W1. The loaded weights are multiplied with the corresponding input data (such as I0 and I1). The results of the multiplication operation are accumulated. The results of the accumulation operation are output to the "OUTPUT" port. From Figure 1A The implementation shown in the figure shows that in the traditional digital computing architecture, the storage module and the computing unit are separated, which requires frequent data movement, resulting in low computing energy efficiency.

[0033] Figure 1B For a typical digital domain CIM computing architecture, Figure 1B As shown in the figure, in the digital computing architecture, the input is loaded from the input cache to the PE (Processing Element) for multiplication and addition operations, that is, the MAC computing unit is embedded in the SRAM (Static Random-Access Memory), and an XNOR (exclusive NOR) gate and a full adder are designed for each SRAM storage unit. The typical representative of the digital domain CIM computing architecture is Column-MAC. Figure 1B Although the digital domain CIM computing architecture shown puts the computing unit and storage together, it will bring a large area overhead. In order to improve this, the related art proposes a simplified design scheme for digital domain CIM, adding only one NOR (NOR) gate to each SRAM storage unit and designing an additional accumulator in the sub-storage array, which greatly improves the area utilization efficiency. This digital domain CIM combines the advantages of digital architecture and CIM, achieves high computing efficiency by integrating calculations into memory, and performs operations through digital logic, avoiding the non-ideality of analog signals, thereby ensuring high precision.

[0034] However, traditionally, digital domain computational memory (CIM) only supports integer type multiplication and addition operations. This limitation makes it difficult to meet the high computational flexibility requirements of modern AI algorithms. In 2023, related technologies successfully achieved the transition from integer computing to floating-point computing by combining digital domain CIM with reconfigurable computing technology and approximate computing methods. This breakthrough significantly improves the computational flexibility of CIM, making the current digital domain CIM more applicable and performant in complex artificial intelligence tasks that require high computational accuracy.

[0035] The above introduces the digital domain CIM computing architecture, and the following introduces the analog domain CIM computing architecture.

[0036] Compared with digital domain CIM, analog domain CIM has higher energy efficiency due to its current or charge-based calculation method, making it more suitable for scenarios that require continuous and uninterrupted operation. Figure 2A and Figure 2B . Figure 2A is a schematic diagram of the simulation domain CIM computing architecture provided in an embodiment of the present application, Figure 2B The embodiment of the present application provides a schematic diagram of a simulation domain CIM architecture based on time calculation. Figure 2A As shown, the analog domain CIM (Computational Memory) computing architecture is based on integrating data processing and computing directly inside the memory unit to eliminate the bandwidth bottleneck and delay of data transmission between the processor and memory in the traditional computing architecture. In the analog domain CIM computing architecture, the memory unit is designed to be able to directly perform computing operations rather than just store data. This is usually achieved by exploiting the laws of physics (such as Kirchhoff's current law) and charge sharing in the memory array to achieve analog computing. In this way, the CIM architecture can create efficient computing primitives, including logic gates and multiplication (MAC) operations.

[0037] Specifically, Figure 2A In the analog domain CIM computing architecture shown, the input signal loaded from the input cache is converted into an analog signal by a DAC (Digital-to-Analog Converter) and enters the memory array. Preferably, the memory array can store weights or other parameters for the selector to read and allocate. Figure 2A In the illustrated embodiment, a successive approximation register (SAR) ADC (analog to digital converter) or a FLASH ADC (flash analog to digital converter) can be used as the core circuit for sampling, quantization and calculation to complete the multiplication and addition calculation in the memory. The SAR ADC performs well in scenarios that require long-term comparison and high-precision quantization (high number of bits). The FLASH ADC is known for its fast conversion speed. It is mainly composed of a comparator, an encoder and a resistor chain, and provides a linear reference voltage for each comparator. However, the power consumption overhead increases with the number of comparators, so it is more suitable for low-bit quantization (<5 bits) scenarios. The design and optimization of ADC is the key to the research of CIM in the analog domain. For example, the related technology proposes a current digital quantization readout circuit with a calculation current tracking function. By sharing the reference current generation circuit and the "three-step quantization" readout acceleration scheme, the area utilization efficiency of the storage array is significantly improved, and the calculation readout access time is shortened.

[0038] As process nodes continue to shrink, circuit operating voltages are gradually reduced, which poses a challenge to the sampling margin of analog computing. In this context, time-domain CIM demonstrates its advantages with its unique computing architecture, see Figure 2B .like Figure 2B As shown in the figure, in the time domain CIM, data is encoded by pulse width or path delay, and the multiplication and addition (MAC) operation is achieved by adjusting the pulse width or measuring the time difference between the rising edges of the pulses. The sampling of the time domain CIM relies on the time-to-digital converter (TDC). Since there are fewer pulse switching activities in the time domain calculation, the energy consumption is lower, which adapts to the development trend of data-intensive applications. The time domain CIM is not only an ideal architecture for realizing energy-saving edge artificial intelligence processors, but also its calculation bit width can be infinitely expanded in theory, and larger-scale data processing can be achieved through wider pulses or more pulses. However, since the time domain calculation needs to be calculated and quantized in the time domain, the calculation speed is relatively slow, especially when the amount of data increases, the calculation time is correspondingly extended, thereby limiting the computing power of the in-memory computing macro unit. Since the delayed signal can be essentially regarded as a continuous analog signal, the time domain CIM can also be regarded as a special analog domain CIM. In the related art, the weight, eigenvalue and MAC operation are tightly integrated in the same macro unit, and the MAC operation is performed using the time domain calculation circuit based on pulse width modulation. Tapeout and verification in a binary weight network showed that the technology achieved a peak energy efficiency of 119.7TOPS / W.

[0039] STT-MRAM (Spin Transfer Torque Magnetic Random Access Memory) is the most mature large-scale integration method for spin electronic devices in the current process, and it has shown significant advantages in non-volatile and high-density storage. However, due to the device characteristics of STT-MRAM itself, it is not suitable for direct use in multi-bit operations. At the same time, reading the accumulated results by magnetoresistive accumulation also requires complex sampling sensing technology. In contrast, SRAM (Static Random Access Memory) achieves storage through voltage, is naturally compatible with digital computing, and exhibits high performance and is easy to work with digital computing units.

[0040] Therefore, exploring the in-memory computing implementation of MRAM (Magnetic Random Access Memory)-SRAM (Static Random Access Memory) hybrid storage has become a feasible computing architecture. This hybrid storage aims to leverage the non-volatility and high-density storage advantages of MRAM to achieve effective storage of weights; at the same time, it uses the high performance of SRAM and its natural compatibility with digital computing units to complete data caching and computing processes. Through this combination, it is expected to achieve an in-memory computing implementation method with low standby power consumption and high computing accuracy, providing new possibilities for future computing system architectures.

[0041] The MRAM and SRAM heterogeneously integrated near-memory computing architecture provided in the embodiments of the present application is intended to deeply tap the huge potential of MRAM in the field of in-memory computing. This architecture not only demonstrates significant technical advantages, but also has extremely high value in practical applications. With the gradual maturity of high-capacity, high-density spin transfer torque magnetic random access memory (STT-MRAM) technology, the low-cost data interaction technology of MRAM-SRAM can be used to integrate large-scale non-volatile storage and main memory on the same chip, thereby making a revolutionary breakthrough in the traditional CPU-DRAM-FLASH computer architecture and giving birth to a new computing-storage integrated architecture.

[0042] Figure 3 is a MRAM-SRAM digital computing circuit diagram provided in an embodiment of the present application, such as Figure 3 As shown, the digital calculation circuit may include: an encoding circuit 301, a weight control circuit 302, a partial product generation circuit 303, a compression circuit 304, and an adder circuit 305, and the digital calculation circuit is set in the MRAM-SRAM memory.

[0043] Among them, the first end of the encoding circuit 301 is a data input end, the second end of the encoding circuit 301 is connected to the first end of the weight control circuit 302 and the first end of the partial product generating circuit 303 respectively, the second end of the partial product generating circuit 303 is connected to the first end of the compression circuit 304; the second end of the compression circuit 304 is connected to the first end of the adder 305; the second end of the adder 305 is the result output end. The encoding circuit 301 is used to encode the input data. Exemplarily, the encoding circuit 301 encodes the input data in a Booth encoding manner, see Table (1). Table (1) is a base 4 Booth encoding table.

[0044] Table (1)

[0045]

[0046]

[0047] The weight control circuit 302 controls the weight data. The weight data and the input data are stored in a magnetic random access memory-static random access memory (MRAM-SRAM) in the form of complement code.

[0048] The partial product generation circuit 303 generates partial products for the encoded input data and the controlled weight data. Booth coding reduces the number of partial products in the multiplication operation by converting the multiplier into a series of special encoding values ​​(such as +1, 0, -1). The partial product is an intermediate result in the multiplication operation, which represents the result of multiplying each bit of the input data by the weight data. The number of bits (bits) of the operands requires the convolution calculation to be completed in multiple cycles. For example, for an 8-bit operand, a 3*3 convolution calculation requires four cycles to complete.

[0049] The compression circuit 304 compresses the partial product results, and the compression circuit may include compressors of different proportions for compressing the partial product results. For example, in each cycle, a compressor tree composed of a mixture of a 4:2 compressor and a 3:2 compressor is used to compress 9 groups of partial product results into 2 groups and send them to the adder 305 to complete the calculation of one cycle. It should be noted that the adder 305 includes a carry-lookahead adder and a shift accumulator. The carry-lookahead adder is a circuit that can quickly complete the addition of multi-bit binary numbers, and it reduces the delay of addition by pre-calculating the carry. Due to the particularity of the convolution operation, the result of each calculation needs to be accumulated on the basis of the result of the previous round. The result of the previous round can be shifted left and added to the result of the current round through the shift accumulator. When the calculation of all cycles is completed, the result in the shift accumulator is the final convolution operation result, which will be output for subsequent processing.

[0050] The above embodiment is illustrative in detail using the operation of a 3*3 convolution kernel of an 8-bit signed number as an example. In practical applications, the above digital computing circuit can be applied to convolution kernel operations of any size, and the embodiments of the present application do not impose any specific limitations.

[0051] The principle of 3:2 compressor is as follows Figure 3A As shown, Figure 3A is a schematic diagram of a 3:2 compressor provided in an embodiment of the present application, such as Figure 3A As shown in FIG. 1 , the 3:2 compressor receives three input operands A, B, and C. An XOR operation is usually used to generate the low-order portion of the sum. Figure 3BIn the 3:2 compressor scenario shown, an XOR operation is used to calculate the partial sum between input operands A, B, and C. For example, an XOR operation may be performed on A and B to obtain an intermediate result, and then this result may be XORed with C to obtain the low-order part of the sum. The generation of a carry signal is an important part of the addition operation. In a 3:2 compressor, a selector may be used to determine whether a carry is generated based on a specific combination of input operands. The selector may select the correct carry signal based on the specific values ​​of A, B, and C combined with the previous intermediate results. The output of a 3:2 compressor generally includes two parts: an equivalent value of the sum (Sum) and a carry signal (Carry). The equivalent value of the sum may be a combination of the low-order part generated by the XOR operation and the carry signal determined by the selector. The carry signal (Carry) is directly output by the selector. By combining the XOR operation and the selector, the 3:2 compressor is able to compress three input operands into two outputs: an equivalent value of the sum (Sum) and a carry signal (Carry).

[0052] Figure 3B is a schematic diagram of a 4:2 compressor provided in an embodiment of the present application, such as Figure 3B The 4:2 compressor shown in Figure 1 accepts four input operands, usually labeled A, B, C, and D. An XOR operation is used to generate the low-order portion of the sum. Figure 3B In the circuit shown, four XOR gates are used to calculate the low-order bits of the sum of the input operands A, B, C, and D. For example, the low-order bits of the sum can be obtained by performing XOR operations on A, B, C, and D in pairs and then XORing the results. The selector may select the correct carry signal based on the specific values ​​of the input operands and the results of previous operations. This usually involves judging different combinations of input operands and outputting the correct carry signal accordingly. Figure 3B The output of the 4:2 compressor shown includes a sum value (Sum) and a carry signal (Cout) and an additional carry signal (Carry). The compressor output result is an equivalent value of the sum (Sum) (possibly the low and high bits) and a possible carry signal. The 4:2 compressor is able to compress four input operands into two equivalent output operands, the sum value (Sum) and the carry signal (Cout).

[0053] Figure 3C Detailed structural diagram of the digital computing circuit provided in the embodiment of the present application. Its working principle is similar to that of the above-mentioned embodiment and will not be described again here.

[0054] The following is an introduction to the complete data writing process of the MRAM-SRAM digital computing circuit provided in the embodiment of the present application, see Figure 4 . Figure 4Schematic diagram of data writing, transfer and calculation waveforms provided in the embodiment of the present application. Figure 4 As shown, for the write operation of MRAM (magnetoresistive random access memory), when the WEB (first control signal) signal is pulled low, the write operation is triggered. Writing MRAM data at a specific address (Address) requires two complete clock cycles.

[0055] In the data transfer phase, when the WEB signal is pulled high, data is transferred from SRAM (static random access memory) to MRAM. This process is divided into three stages, each of which is triggered by an edge of the clock signal (CLK). Therefore, it takes two clock cycles to complete a complete data transfer. Due to the capacity limitation of each SRAM sub-array, this data transfer process needs to be repeated three times to fill it up.

[0056] The calculation process is controlled by the CIMEN (second control signal) signal, and its priority is higher than the WEB signal. When the CIMEN signal is at a high level, the calculation unit will start to perform the calculation task. Exemplarily, each calculation task lasts for four clock cycles, and between each calculation, the CIMEN signal is pulled low to reload the input data and switch the weight data. This design ensures the continuity and accuracy of the calculation process.

[0057] The following describes an MRAM near-memory convolution operation circuit including the above-mentioned MRAM_SRAM heterogeneous integrated digital computing circuit. Figure 5 , Figure 5 Schematic diagram of a near-memory convolution operation circuit of an MRAM-SRAM hybrid integrated integer quantization convolution operation in-memory computing unit provided in an embodiment of the present application. The MRAM near-memory convolution operation circuit is an MRAM-SRAM hybrid integrated integer quantization convolution operation in-memory computing unit (Integer Quantization Convolution CIM, IQC-CIM). Figure 5 As shown, the logic control circuit, memory array, register group and accumulation calculation unit. Figure 5As shown, the near-memory convolution operation circuit includes: a logic control circuit 501, a storage array 502, a register group 503 and a multiplication-accumulation calculation unit 504. The top-level logic control circuit 501 of the IQC-CIM architecture includes: a decoding circuit, a control circuit, and a level shifter. The decoding circuit is responsible for converting the input instructions or data into a format that the calculation unit can understand; the control circuit guides the operation of each part of the calculation unit according to the converted information; the level shifter handles the data conversion between different voltage levels to ensure that the data is correctly transmitted between different modules. The storage array is divided into multiple efficient MRAM-SRAM sub-array combinations. All sub-arrays share a set of logic control circuits to maximize the utilization of resources. Among them, MRAM is used to store weight data, while SRAM is used as an efficient cache mechanism to extract the weight data and convert it into a voltage signal to provide the digital operation circuit for MAC (multiplication-accumulation) operation. Among them, the multiplication-accumulation calculation unit 504 includes: a multiplier and an accumulator. In order to ensure that the data flow between sub-arrays is unimpeded, a driving compensation circuit is introduced between the sub-arrays. As shown Figure 5 As shown, the multiplication and accumulation calculation unit is set on the side close to the memory array to complete the multiplication and accumulation operation of the input data and the weight data. The weight data is stored in the MRAM, and the weight data is transferred from the MRAM to the SRAM, and then enters the register group through the SRAM; the input data is directly input into the register group. Then the weight data and the input data enter the multiplication and accumulation calculation unit to complete the multiplication and accumulation calculation.

[0058] It should be noted that the weight data enters the register group in order to have a clock cycle buffer to facilitate subsequent calculation operations.

[0059] exist Figure 5 In the illustrated embodiment, the SRAM subarray is designed to store multiple convolution kernels (weight data) simultaneously, so that different convolution kernels can be quickly loaded from the SRAM when performing continuous convolution operations without frequent reading from the MRAM.

[0060] In the register group, only one convolution kernel and its corresponding input data are stored at a time to ensure that a convolution operation is completed within a single clock cycle. In this way, the system can efficiently perform continuous convolution operations, thereby improving the overall processing speed and efficiency.

[0061] For example, three weight data can be stored in the SRAM subarray, and each operation involves a 3x3 convolution operation, which means that a 72-bit convolution kernel (corresponding to a 3x3 weight matrix) will be convolved with 72 bits of input data.

[0062] Figure 5In the illustrated embodiment, an efficient 3x3 convolution operation can be implemented, supporting 8-bit input data, 8-bit weight, and generating 20-bit output. Its core storage array uses 2T2M (two transistors and two magnetic tunnel junctions) storage cells to ensure efficient storage and access of data. When writing data, the module requires a voltage of 1.8V; when reading data, only a voltage of 0.9V is required, which effectively reduces energy consumption. A complete read-write functional unit is defined as a "bank", and each bank contains a storage cell array and necessary peripheral circuits. Figure 5 In the embodiment shown, a total of 20 banks are integrated, and each bank is equipped with 9 arrays to support large-scale parallel data processing. Preferably, 28nm CMOS (Complementary Metal-Oxide-Semiconductor) and 70nm MTJ (Magnetic Tunnel Junction) processes are selected to ensure high integration and performance of the module. Each array contains 128*64 effective MTJ units, with a total area of ​​approximately 1146 square microns and a storage capacity of 64*64=4Kb. Therefore, the total storage capacity of the entire module is 20*9*64*64=720Kb, which meets the high demand for storage capacity for complex convolution operations.

[0063] The following is an introduction to the unit layout of IQC-CIM, see Figure 6 . Figure 6 Schematic diagram of the unit layout of the IQC-CIM provided in the embodiment of the present application. Figure 6 As shown in FIG. 1 , the unit integrates all necessary peripheral circuits, a storage array, and nine groups of MRAM-SRAM subarrays. The combined storage capacity of these subarrays reaches 36Kb of MRAM and 0.281Kb of SRAM, fully meeting the needs of high-performance data processing. At the same time, the IQC-CIM provided by the embodiment of the present application makes the area overhead of the entire unit only 0.0575mm 2 , which not only improves the data processing capability, but also reduces the manufacturing cost and physical space occupation through a compact layout. See Table (2) for details. Table (2) is a parameter table of the MRAM-SRAM hybrid storage and computing macro unit provided in the embodiment of the present application.

[0064] Table (2)

[0065]

[0066] The computing architecture proposed in the embodiment of the present application significantly reduces power consumption by avoiding unnecessary data transfer operations in the von Neumann architecture. The architecture uses the heterogeneous integration technology of SRAM-MRAM to store weights, thereby improving data access efficiency and storage density. In addition, the near-memory multiplier design combined with the compressor and adder further enhances the computing power while optimizing the area and speed performance. See Table (3), which is a comparison of the computing architecture proposed in the embodiment of the present application with respect to other related technologies. It can be seen that the computing architecture proposed in the embodiment of the present application has obvious advantages in power consumption, area and speed.

[0067] Table (3)

[0068]

[0069] In some embodiments, the present application also provides a VGG convolutional neural network model including 5 convolutional layers and 3 fully connected integer quantization layers for neural network. Figure 7 The near-memory convolution operation circuit provided in the embodiment of the present application can be used in a convolutional neural network, and the convolutional neural network completes the multiplication and accumulation operation of the convolutional neural network according to the above-mentioned near-memory convolution operation circuit. Figure 7 Schematic diagram of the VGG convolutional neural network model provided in the embodiment of the present application. After training and testing on the CIFAR-10 dataset, the model showed an accuracy of up to 93.2%.

[0070] In some embodiments, the near-memory convolution operation circuit provided in the embodiments of the present application can be used in a chip system, and the chip system is used to complete the multiplication and accumulation operations of the chip system according to the above-mentioned near-memory convolution operation circuit.

[0071] In some other embodiments, the near-memory convolution operation circuit provided in the embodiments of the present application can be used in an electronic device, and the electronic device performs a multiplication-accumulation operation on the electronic device according to the near-memory convolution operation circuit.

[0072] In summary, the embodiment of the present application provides a near-memory convolution operation circuit that can improve computing efficiency and reduce power consumption. Figure 8A to Figure 8B .

[0073] Furthermore, the power consumption analysis of the near-memory convolution operation circuit proposed in the embodiment of the present application is carried out, see Fig. 8A . Fig. 8A Schematic diagram of the relationship between power consumption per bit of data transfer and VDD provided in the embodiment of the present application. Fig. 8AThe relationship between the power consumption per bit of data transfer between MRAM and SRAM and the power supply voltage is shown. Under the TT process corner, when the power supply voltage is 0.7V, the power consumption per bit of data transfer is 30.3fJ. However, with the increase of voltage, especially under the FF process corner, the increase of power consumption becomes more significant. The simulation results show that when the storage array operates in the voltage range of 0.6-0.8V, better energy efficiency performance can be obtained. In a single MRAM-SRAM hybrid computing unit, the SRAM can store 4 groups of 3*3 integer quantized convolution kernels. Based on this feature, 4 data transfers to fill up the SRAM and the subsequent 4 convolution operations are regarded as a complete data transfer-computation process.

[0074] At a supply voltage of 0.7V and a clock cycle of 100MHz, the entire 72 operations consume only 11.52pJ of energy, achieving an energy efficiency of up to 6.25TOPS / W. Figure 8B , Figure 8B Schematic diagram of the power consumption ratio of each part in the data transfer-calculation process provided by the embodiment of the present application. Figure 8B As shown in the figure, the storage array accounts for the highest proportion of power consumption, reaching 58%, the computing circuit accounts for 27%, and the clock and control circuit account for the remaining 15%. It should be noted that when the data in the SRAM does not need to be updated and only the convolution operation between the input data and the fixed 4 sets of convolution kernels is performed, the power consumption overhead caused by data transfer will become negligible as the number of calculations increases. In this case, the energy efficiency can be further improved to an amazing 14.9TOPS / W. "

[0075] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0076] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0077] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0078] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0079] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0080] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0081] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0082] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0083] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A near-memory convolution operation circuit, characterized in that: The near-memory convolution operation circuit is used to perform multiplication and accumulation operations; The digital near-storage convolution operation circuit comprises: a logic control circuit, a memory array, a register group and a multiplication and accumulation calculation unit; The multiplication-accumulation calculation unit comprises: a multiplier and an accumulator; The memory array includes a magnetic random access memory subarray and a static random access memory subarray; The multiplication-accumulation calculation unit is arranged at a side close to the memory array; The register group is located between the multiplication-accumulation calculation unit and the memory array.

2. The near-memory convolution operation circuit according to claim 1, characterized in that: The static random access memory loads weight data from the magnetic random access memory, and the static random access memory stores a plurality of different weight data simultaneously.

3. The near-memory convolution operation circuit according to claim 1, characterized in that: The input data is directly input into the register group, and the register group stores the weight data corresponding to the input data one at a time.

4. The near-memory convolution operation circuit according to claim 1, characterized in that: The multiplication-accumulation calculation unit is realized by a digital calculation circuit, wherein the digital calculation circuit comprises: an encoding circuit, a weight control circuit, a partial product generation circuit, a compression circuit and an adder circuit; The encoding circuit is used to encode the input data; The weight control circuit is used to control the weight data; The partial product generating circuit is used to generate a partial product result for the encoded input data and the controlled weight data; The compression circuit includes compressors of different ratios for compressing the partial product results; The adder circuit includes a carry adder and a shift accumulator, which are used to complete multiplication and accumulation calculations.

5. The near-memory convolution operation circuit according to claim 4, characterized in that: The weight data is stored in the magnetic random access memory in the form of complement code.

6. The near-memory convolution operation circuit according to claim 4, characterized in that: The encoding circuit performs Booth encoding on the input data.

7. The near-memory convolution operation circuit according to claim 4, characterized in that: A first control signal controls data reading of the static random access memory and data writing of the magnetic random access memory; When the first control signal is at a low level, writing the weight data into the magnetic random access memory; When the first control signal is at a high level, the static random access memory reads the weight data from the magnetic random access memory; The second control signal controls the digital computing circuit to enter into computing operation.

8. The near-memory convolution operation circuit according to claim 7, characterized in that: The priority of the second control signal is higher than the priority of the first control signal.

9. A convolutional neural network, characterized in that: It includes a near-memory convolution operation circuit as described in any one of claims 1-8, and the convolutional neural network is used to complete the multiplication and accumulation operation of the convolutional neural network according to the near-memory convolution operation circuit.

10. A chip system, characterized in that: It includes a near-memory convolution operation circuit as described in any one of claims 1-8, and the chip system is used to complete the multiplication and accumulation operation of the chip system according to the near-memory convolution operation circuit.

Images