A memory-computing integrated computing system and method supporting deep convolution channel full parallel calculation and a memory-computing integrated chip

Abstract

This invention provides an in-memory computing system, method, and chip that supports fully parallel computation of deep convolution channels. The system includes: a parallel computing module comprising k×k multi-bit data units, each multi-bit data unit comprising q storage sub-units for storing k×k q-bit activation value storage data, and performing simulated multiplication and accumulation operations on the k×k q-bit activation value storage data and input weight parameters of size (k×k,1); a weight configuration circuit connected to the parallel computing module, used to perform bit-level simulation recombination of the q-bit weights in the simulated multiplication and accumulation operation output by the parallel computing module, obtaining a simulated multiplication and accumulation result of 1-bit input weight parameters and q-bit activation value storage data; an ADC quantization circuit connected to the weight configuration circuit, used to quantize the simulated multiplication and accumulation result, obtaining integer data output; and an activation value update circuit connected to the bit line and read bit line of the column where the parallel computing module is located, used to perform local cyclic updates of the activation value storage data within the parallel computing module. This invention can increase the parallelism of array computation and realize local cyclic updates of activation values ​​within the array, thereby improving energy efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an in-memory computing system, method and chip that supports fully parallel computation of deep convolution channels. Background Technology

[0002] In the hardware design for accelerating neural network models, traditional computing architectures separate storage and computation. When faced with neural network computations with a large number of parameters, most of the power consumption is consumed in the process of moving computation parameters from off-chip storage to the processor computing unit.

[0003] In-memory computing chips enable storage and computing capabilities, greatly reducing the need to move parameters between storage and computing. They are an excellent solution for edge computing with low power consumption, low latency, and small area requirements.

[0004] However, depthwise separable networks replace the traditional k×k multiply-accumulate operation of N channels in convolution with a k×k multiply-accumulate operation in depthwise convolution, reducing the accumulation length of the multiply-accumulate operation by a factor of N. Furthermore, because the convolution window needs to move across the input feature map, the activation values, fixed in the in-memory array, require switching and updating operations. This means that traditional in-memory arrays can only perform calculations row by row during computation, and the activation value data is reloaded from off-chip storage in each cycle, resulting in low energy efficiency. Summary of the Invention

[0005] This invention provides an in-memory computing system, method, and chip that supports fully parallel computation of deep convolution channels, in order to solve the problem that existing technologies can only perform computation in a single line and reload activation value data from off-chip storage in each cycle of computation, resulting in low energy efficiency.

[0006] This invention provides an in-memory computing system that supports fully parallel computation of deep convolution channels.

[0007] According to one aspect of the present invention, the in-memory computing system supporting fully parallel computation of depthwise convolution channels includes:

[0008] The parallel computing module includes k×k multi-bit data units, each of which includes q storage sub-units. Each storage sub-unit is used for writing, reading, and calculating 1-bit stored data, while the multi-bit data units are used for synchronous writing, reading, and calculating q-bit stored data. The parallel computing module stores k×k q-bit activation value storage data and performs simulated multiplication and accumulation operations with input weight parameters of size (k×k,1), where k and q are both positive integers greater than or equal to 2.

[0009] A weight configuration circuit, connected to the parallel computing module, is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module, to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data.

[0010] An ADC quantization circuit, connected to the weight configuration circuit, is used to quantize the multiply-accumulate simulation result to obtain an integer data output.

[0011] An activation value update circuit is provided, which connects the bit line and the read bit line of the column where the parallel computing module is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

[0012] According to another aspect of the present invention, the in-memory computing system supporting fully parallel computation of deep convolution channels includes:

[0013] There are M×N parallel computing modules, each of which includes k×k multi-bit data units, and each multi-bit data unit includes q storage sub-units. The storage sub-units are used for writing, reading, and calculating 1-bit stored data, and each multi-bit data unit is used for synchronous writing, reading, and calculating q-bit stored data. The parallel computing modules are used to store q-bit activation value storage data of size k×k, and to perform simulated multiplication and accumulation operations on the q-bit activation value storage data of size k×k and input weight parameters of size (k×k,1), where M and N are positive integers greater than or equal to 1, and k and q are positive integers greater than or equal to 2.

[0014] A weight configuration circuit, connected to the parallel computing module, is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module, to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data.

[0015] An ADC quantization circuit, connected to the weight configuration circuit, is used to quantize the multiply-accumulate simulation result to obtain an integer data output.

[0016] An activation value update circuit is provided, which connects the bit line and the read bit line of the column where the parallel computing module is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

[0017] According to the in-memory computing system that supports fully parallel computation of deep convolution channels provided by the present invention, the storage sub-units are connected to the same computation bit line.

[0018] According to the in-memory computing system that supports fully parallel computation of deep convolution channels provided by the present invention, the storage sub-unit is an embedded dynamic random access memory (eDRAM) unit.

[0019] According to the in-memory computing system supporting fully parallel deep convolution channels provided by the present invention, the eDRAM unit includes:

[0020] A first transistor, wherein a first terminal of the first transistor is connected to a bit line, and a gate of the first transistor is connected to a write word line;

[0021] The second transistor has its gate connected to the second terminal of the first transistor, and the second terminal of the second transistor is connected to a power supply.

[0022] A third transistor, wherein the gate of the third transistor is connected to the second terminal of the first transistor, the first terminal of the third transistor is connected to the first input signal, and the second terminal of the third transistor is connected to the first terminal of the second transistor;

[0023] A fourth transistor, wherein the first terminal of the fourth transistor is connected to the first terminal of the second transistor and the second terminal of the third transistor; the second terminal of the fourth transistor is connected to the read word line; and a second memory node is provided between the first terminal of the fourth transistor and the first terminal of the second transistor and the second terminal of the third transistor.

[0024] The fifth transistor has a first terminal connected to a power supply, a second terminal connected to the second terminal of the third transistor, and a gate connected to a second input signal, wherein the second input signal is the opposite of the first input signal.

[0025] A first capacitor is disposed between the connection line between the gate of the third transistor and the gate of the second transistor and the second terminal of the first transistor, and is used to store data received by a first storage node. The first storage node is disposed on the upper plate of the first capacitor and is used to receive data written by the first transistor.

[0026] The second capacitor is disposed between the second terminal of the fifth transistor and the calculation bit line, and is used to output the voltage change on the upper plate of the second capacitor to the calculation bit line. The voltage change on the upper plate is obtained according to the data input by the fifth transistor or the data stored in the first storage node.

[0027] According to the in-memory computing system supporting fully parallel deep convolution channels provided by the present invention, the eDRAM unit includes:

[0028] A sixth transistor, wherein the first terminal of the sixth transistor is connected to a bit line, and the gate of the sixth transistor is connected to a first read word line;

[0029] The seventh transistor has its first terminal connected to the first terminal of the sixth transistor and then connected to the same bit line as the first terminal of the sixth transistor. The gate of the seventh transistor is connected to the second read word line.

[0030] The eighth transistor and the ninth transistor form an inverter, and one end of the inverter is connected to the second end of the sixth transistor and the second end of the seventh transistor.

[0031] The third capacitor is disposed between the second terminal of the sixth transistor and the second terminal of the seventh transistor and the first terminal of the inverter, and is used to store the data received by the third storage node. The third storage node is disposed on the upper plate of the third capacitor and is used to receive the data written by the sixth transistor and the seventh transistor.

[0032] A tenth transistor, the first terminal of which is connected to the second terminal of the inverter, the second terminal of which is connected to the read bit line, the gate of which is connected to the read word line, and a fourth memory node is provided between the tenth transistor and the second terminal of the inverter;

[0033] The eleventh transistor, the first terminal of which is connected to the POS terminal, and the gate of which is connected to the third memory node;

[0034] The twelfth transistor has its first terminal connected to the second terminal of the eleventh transistor, its gate connected to the fourth storage node, the fourth storage node being disposed between the inverter and the first terminal of the tenth transistor, and its second terminal connected to the NEG terminal.

[0035] The fourth capacitor has its upper plate connected to the second terminal of the eleventh transistor and the first terminal of the twelfth transistor, and its lower plate connected to the calculation bit line.

[0036] According to the in-memory computing system that supports fully parallel computation of deep convolution channels provided by the present invention, the activation value update circuit is configured as a tri-state inverter, and the tri-state inverter is connected to the bit line and read bit line of the column where the parallel computing module is located.

[0037] This invention also provides a memory-based computing method that supports fully parallel computation of depthwise convolution channels. The method is used for depthwise convolution computation in depthwise separable neural networks. The method is implemented using a memory-based computing system that supports fully parallel computation of depthwise convolution channels as described in any of the preceding claims. The method includes:

[0038] To obtain the input feature maps of each input channel in a deep separable neural network to be calculated by depthwise convolution, the deep separable neural network includes M input channels, and each input channel is divided into N sub-feature maps of size k×k;

[0039] The input feature map is divided into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of first columns and then rows;

[0040] The N-row parallel computing module stores the N k×k q-bit activation values ​​in the input channel.

[0041] The input weight parameters of size (k×k,1) are multiplied and accumulated with the N k×k q-bit activation values ​​stored in the column of the input feature map to obtain the N×q multiply-accumulate results output by the input channel.

[0042] The N×q multiply-accumulate results output from the input channel are simulated and weighted by q bits using a weight configuration circuit to obtain N multiply-accumulate results.

[0043] The N multiplication and accumulation calculation results are quantized and output using an ADC quantization circuit.

[0044] Based on the sliding direction of the convolution kernel in the input feature map, the activation value update circuit performs a local cyclic update of the activation value storage data inside the parallel computing module.

[0045] The in-memory computing method supporting fully parallel computation of depthwise convolution channels provided by the present invention further includes:

[0046] The step of performing a local cyclic update of the activation value storage data within the parallel computing module based on the sliding direction of the convolution kernel in the input feature map, through the activation value update circuit, specifically includes:

[0047] As the convolutional kernel slides vertically in the input feature map, the activation value update circuit updates the activation values ​​of adjacent k columns in each cycle.

[0048] As the convolutional kernel slides horizontally across the input feature map, the activation value update circuit updates the activation values ​​every k columns in each cycle.

[0049] The in-memory computing method supporting fully parallel computation of depthwise convolution channels provided by the present invention further includes:

[0050] When the i-th input channel performs multiplication-accumulation calculation, the (i+1)-th input channel performs a convolution kernel sliding operation; or,

[0051] When the (i+1)th input channel performs multiply-accumulate calculation, the ith input channel performs a convolution kernel sliding operation;

[0052] Where 1≤i≤M.

[0053] The present invention also provides a memory computing chip, the memory computing chip comprising a memory computing system as described in any of the preceding claims that supports fully parallel computation of deep convolution channels.

[0054] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the in-memory computing method supporting fully parallel computation of deep convolution channels as described in any of the preceding claims.

[0055] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the in-memory computing method supporting fully parallel computation of deep convolution channels as described in any of the preceding claims.

[0056] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the in-memory computing method that supports fully parallel computation of deep convolution channels as described in any of the preceding claims.

[0057] The present invention provides an in-memory computing system, method and in-memory computing chip that supports fully parallel computing of deep convolution channels. It increases the parallelism of array computing through parallel computing modules, achieves efficient updates that match the movement of convolution windows by using an activation value update method adapted to deep convolution computing, and reduces array update power consumption by reducing the update frequency of array data, thereby improving energy efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the structure of an in-memory computing system that supports fully parallel computation of deep convolution channels, provided in an embodiment of the present invention.

[0060] Figure 2 This is a structural diagram of the 5T2C storage unit provided in an embodiment of the present invention;

[0061] Figure 3 This is a schematic diagram of the structure of a multi-bit data unit provided in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the working timing of a multi-bit data unit provided in an embodiment of the present invention;

[0063] Figure 5 This is a structural diagram of the 7T2C storage unit provided in an embodiment of the present invention;

[0064] Figure 6 This is a flowchart illustrating the in-memory computing method that supports fully parallel computation of depthwise convolution channels, as provided in an embodiment of the present invention.

[0065] Figure 7 This is a diagram of a computational mapping architecture that supports fully parallel computation of depth-separable network channels, as provided in an embodiment of the present invention.

[0066] Figure 8 A flowchart illustrating the activation value local cyclic update method adapted to a fully parallel computing architecture provided in this embodiment of the invention;

[0067] Figure 9 A schematic diagram of the pipeline timing design for depthwise convolution computation and memory update provided in an embodiment of the present invention;

[0068] Figure 10 This is the second schematic diagram of the structure of the in-memory computing chip provided in the embodiment of the present invention;

[0069] Figure 11 This diagram illustrates the energy and area distribution of the in-memory computing chip provided in this embodiment of the invention when performing depthwise separable convolution calculations.

[0070] Figure 12 This is a comparison of normalized update power consumption of different architectures during deep separable neural network computation, provided by an embodiment of the present invention.

[0071] Figure 13 A performance comparison chart of the in-memory computing chip provided in this embodiment of the invention with other advanced technologies;

[0072] Figure 14 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0074] In real-world applications, with the rapid development of artificial intelligence, a large number of different types of neural network models have emerged. To meet diverse application needs and adapt to various complex application scenarios, the parameters of neural network models are becoming increasingly large. However, such large models pose a burden on the implementation of edge AI. Edge AI applications have requirements for low power consumption, low latency, and small area, and edge devices have limited storage resources, making large models unsuitable for localized model inference. Therefore, compressed models are more suitable for local deployment. Quantized deep separable neural network models are a type of compressed model. They use deep separable convolutions instead of traditional convolution calculations, reducing the number of model parameters while maintaining inference accuracy. Furthermore, in the hardware design for accelerating neural network models, because traditional computing architectures separate storage and computation, most of the power consumption for neural network computations with a large number of parameters is consumed in the process of moving computational parameters from off-chip storage to the processor computing unit. In-memory computing chips provide storage and computation capabilities, greatly saving the transfer of parameters between storage and computation, making them an excellent solution for the low power consumption, low latency, and small area requirements of edge computing. However, deploying deep separable networks in in-memory computing chips faces many energy efficiency limitations.

[0075] The core of a deep separable neural network is deep separable convolution, which consists of depthwise convolution and pointwise convolution. During deep convolution computation, a k×k kernel is configured for each input channel, and the deep convolutional layer corresponding to N input channels has N k×k convolutions. Because each k×k convolution corresponds to a different activation value of the input feature map, compared to traditional convolutional neural network operations with N input channels, M input channels, and k×k convolutions, the reuse rate of input activation values ​​is reduced by a factor of M. This results in a decrease in throughput by a factor of M when deploying this network in a traditional in-memory computing chip array, without a reduction in power consumption, thus reducing overall energy efficiency by a factor of M. This is because in deep separable neural networks, model weight data is reused more times than input activation values, so fixing the input activation values ​​in the in-memory computing array has a greater energy efficiency advantage. This is because the same weight input can perform more computations with the input activation values, achieving higher parallelism, thereby improving throughput and energy efficiency.

[0076] However, even if the activation values ​​of a deep separable network are fixed in a traditional in-memory array, its energy efficiency is still low. This is because deep separable networks replace the traditional N-channel k×k multiply-accumulate operation of convolution with a k×k multiply-accumulate operation of depthwise convolution, reducing the accumulation length of the multiply-accumulate operation by a factor of N. Furthermore, because the convolution window needs to move across the input feature map, fixing the activation values ​​in the in-memory array requires switching and updating operations, forcing the traditional in-memory array to perform calculations only one row at a time, resulting in low energy efficiency. Therefore, a reasonable mapping method to achieve full reuse of weight data within a channel can improve the computational parallelism of the in-memory array, thereby improving energy efficiency.

[0077] However, once the input activation values ​​are fixed in the in-memory array, the operation of updating the activation values ​​with the convolution window is inevitable. In traditional designs, data is reloaded from off-chip memory in each computation cycle. This operation consumes a lot of energy, so updating the activation values ​​directly on-chip through simple shift operations is a more efficient approach.

[0078] Deeply separable neural networks can significantly reduce the parameter size of neural network models while ensuring model accuracy. However, when deployed for inference in in-memory computing chips, their computational performance is limited by the following: (1) The input data reuse rate of deep convolution computation is much smaller than that of traditional convolution computation, resulting in low array utilization and insufficient power consumption optimization of in-memory computing chips; (2) The accumulation length of the multiply-accumulate operation in deep convolution computation is much smaller than that of traditional convolution, resulting in low computational parallelism of the in-memory computing array; (3) To match the movement of the convolution window in the feature map, the power consumption for updating the array stored in the in-memory computing chip is high, thus failing to achieve the ideal performance of the in-memory computing chip.

[0079] To address the aforementioned energy efficiency limitations, this invention proposes a computing system that supports fully parallel computation of deep separable network channels. This invention relates to the field of artificial neural network technology and in-memory computing based on embedded dynamic random access memory (eDRAM). More specifically, it relates to a hardware implementation of an eDRAM-based in-memory computing multiply-accumulate unit structure for accelerating deep separable neural networks. The proposed computing system, supporting fully parallel computation of deep separable network channels, increases array computational parallelism, thereby improving energy efficiency. This invention also provides an activation value update scheme adapted for deep convolution computation, achieving efficient updates that match convolution window movement. By reducing the update frequency of array data, array update power consumption is reduced, thus improving energy efficiency. Furthermore, this invention provides a pipelined timing design for deep convolution computation and memory updates, designing the multiply-accumulate (MAC) computation and convolution sliding window movement operations in a pipelined manner, improving system throughput.

[0080] Specifically, on the one hand, embodiments of the present invention provide an in-memory computing system that supports fully parallel computation of deep convolution channels.

[0081] As an implementation method of an in-memory computing system that supports fully parallel computation of deep convolution channels, specifically, Figure 1 This is a schematic diagram of the in-memory computing system supporting fully parallel computation of depthwise convolution channels provided in an embodiment of the present invention, as shown below. Figure 1As shown, the in-memory computing system supporting fully parallel computation of deep convolution channels includes a parallel computing module 100. The parallel computing module 100 includes k×k multi-bit data units, each of which includes q storage sub-units. The storage sub-units are used for writing, reading, and computing 1-bit stored data, and the multi-bit data units are used for synchronous writing, reading, and computing q-bit stored data. The parallel computing module is used to store q-bit activation value storage data of size k×k, and to perform simulated multiplication and accumulation operations on the k×k q-bit activation value storage data and the input weight parameters of size (k×k,1) vector, where k and q are both positive integers greater than or equal to 2.

[0082] A weight configuration circuit 200 is connected to the parallel computing module 100 and is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data.

[0083] An ADC quantization circuit 300 is connected to the weight configuration circuit 200 and is used to quantize the multiply-accumulate simulation result to obtain an integer data output.

[0084] An activation value update circuit 400 is connected to the bit line and read bit line of the column where the parallel computing module 100 is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

[0085] As another implementation of an in-memory computing system supporting fully parallel computation of deep convolution channels, the in-memory computing system supporting fully parallel computation of deep convolution channels includes M×N parallel computing modules. Each parallel computing module includes k×k multi-bit data units, and each multi-bit data unit includes q storage sub-units. The storage sub-units are used for writing, reading, and computing 1-bit stored data, and each multi-bit data unit is used for synchronous writing, synchronous reading, and synchronous computation of q-bit stored data. The parallel computing modules are used to store q-bit activation value storage data of size k×k, and perform simulated multiplication and accumulation operations on the q-bit activation value storage data of size k×k and input weight parameters of size (k×k,1), where M and N are positive integers greater than or equal to 1, and k and q are positive integers greater than or equal to 2.

[0086] A weight configuration circuit, connected to the parallel computing module, is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module, to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data.

[0087] An ADC quantization circuit, connected to the weight configuration circuit, is used to quantize the multiply-accumulate simulation result to obtain an integer data output.

[0088] An activation value update circuit is provided, which connects the bit line and the read bit line of the column where the parallel computing module is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

[0089] Optionally, the storage sub-unit can be an enhanced dynamic random access memory (eDRAM) cell, and the cell architecture of each eDRAM storage sub-unit can be a 5T2C or 7T2C structure.

[0090] Optionally, the storage sub-cells are connected to the same computation bit line to accumulate the computation results of each storage sub-cell.

[0091] Based on the above embodiments, by way of example, refer to Figure 1 The in-memory computing chip includes an in-memory computing system that supports fully parallel computation across deep convolution channels. This system comprises multiple parallel computing modules, a weight configuration circuit, an ADC quantization circuit, and an activation value update circuit. Each parallel computing module includes multiple multi-bit data units, and each multi-bit data unit includes multiple storage sub-units. The unit architecture of each storage sub-unit can be selected as either a 5T2C or 7T2C unit.

[0092] Optionally, the 5T2C storage subunit provided in this embodiment of the invention includes:

[0093] A first transistor, wherein a first terminal of the first transistor is connected to a bit line, and a gate of the first transistor is connected to a write word line;

[0094] The second transistor has its gate connected to the second terminal of the first transistor, and the second terminal of the second transistor is connected to a power supply.

[0095] A third transistor, wherein the gate of the third transistor is connected to the second terminal of the first transistor, the first terminal of the third transistor is connected to the first input signal, and the second terminal of the third transistor is connected to the first terminal of the second transistor;

[0096] A fourth transistor, wherein the first terminal of the fourth transistor is connected to the first terminal of the second transistor and the second terminal of the third transistor; the second terminal of the fourth transistor is connected to the read word line; and a second memory node is provided between the first terminal of the fourth transistor and the first terminal of the second transistor and the second terminal of the third transistor.

[0097] The fifth transistor has a first terminal connected to a power supply, a second terminal connected to the second terminal of the third transistor, and a gate connected to a second input signal, wherein the second input signal is the opposite of the first input signal.

[0098] A first capacitor is disposed between the connection line between the gate of the third transistor and the gate of the second transistor and the second terminal of the first transistor, and is used to store data received by a first storage node. The first storage node is disposed on the upper plate of the first capacitor and is used to receive data written by the first transistor.

[0099] The second capacitor is disposed between the second terminal of the fifth transistor and the calculation bit line, and is used to output the voltage change on the upper plate of the second capacitor to the calculation bit line. The voltage change on the upper plate is obtained according to the data input by the fifth transistor or the data stored in the first storage node.

[0100] Furthermore, in the storage write phase of the above-mentioned 5T2C structure, the first transistor and the first capacitor form a write circuit. Both the first input signal and the second input signal are set to a high level, the second storage node is kept at a high level, the first terminal of the first transistor receives the data to be written, when the write word line is set to a high level, the first transistor is turned on, the data to be written is written to the first storage node through the first transistor, when the write word line is set to a low level, the first transistor is turned off, and the write operation ends.

[0101] In the storage and read phase of the 5T2C structure described above, the second and third transistors form a first inverter, which, together with the fourth transistor, forms a read circuit. The first input signal is set to a low level, and the second input signal is set to a high level. The data stored in the second storage node is the inverted result of the data stored in the first storage node. When the read word line is set to a high level, the fourth transistor is turned on, and the data stored in the second storage node is read onto the read bit line through the fourth transistor. When the read word line is set to a low level, the fourth transistor is turned off, and the read operation ends.

[0102] During the storage read phase, a second inverter is provided between the second terminal of the fourth transistor and the read bit line. The second inverter is used to invert the data read by the fourth transistor. The fifth transistor is turned off to avoid corrupting the read data.

[0103] In the computation phase of the above 5T2C structure, the fifth transistor and the second capacitor form a computation circuit, the write line and the read line are set to a low level, and the computation phase includes a pre-charge phase and a capacitive coupling phase.

[0104] Specifically, during the precharge phase, the first input signal is set to a high level, the second input signal is set to a low level, the second storage node is precharged to a high level, and the computation bit line is precharged to a high level;

[0105] During the capacitive coupling phase, the gate of the fifth transistor receives 1 bit of input data. When the 1 bit of input data is 1, the first input signal is set to a low level, the second input signal is set to a high level, the fifth transistor is turned off, and the second storage node reads the data stored in the first storage node. When the data read by the second storage node is 1, the second storage node changes from a high level to a low level, and the voltage change of the second storage node is coupled to the calculation bit line through the second capacitor. When the data read by the second storage node is 0, there is no voltage change in the second storage node.

[0106] During the capacitive coupling stage, the gate of the fifth transistor receives 1 bit of input data. When the 1 bit of input data is 0, the first input signal is set to a high level, the second input signal is set to a low level, the fifth transistor is turned on, and the second storage node has no voltage change.

[0107] Specifically, Figure 2 A structural diagram of the 5T2C storage subunit provided in an embodiment of the present invention is shown below. Figure 2 As shown, this unit can be divided into three parts: storage writing, storage reading, and calculation. Transistor M1 and the node parasitic capacitance CMOS implement storage writing; the data to be written is stored in node Q via the bit line (BL) through M1. Transistors M2, M3, and M4 implement storage reading; the data stored at point Q is inverted by M2 and M3 and output via the read bit line (RBL) through M4. Note that to ensure consistency of the stored data in the output data domain, an inverter needs to be connected after RBL. Transistor M5 and the metal capacitor CMOM implement calculation; the activation value is input from M5, and the voltage stored on the upper plate of CMOM is affected by the storage weight at point Q, thus completing the multiplication calculation. The resulting simulation results are accumulated on the compute bit line (CBL).

[0108] The core structure of the chip consists of eight 5T2C cells on the same row of the eDRAM array: Multi-bit Data Unit (MbDU). Figure 3 This is a schematic diagram of the structure of a multi-bit data unit provided in an embodiment of the present invention, as shown below. Figure 3 As shown, each row of the array includes multiple MbDUs, where the number of MbDUs represents the number of output channels for the weights of the neural network model deployed in the eDRAM memory chip. The input to each row performs computational operations with all the output channels stored in that row. Figure 4 This is a timing diagram illustrating the operation of a multi-bit data unit provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the working timing of this structure is explained by taking the first unit from the left that stores bit "1" and the second unit that stores bit "0" as an example. The working timing of this structure includes three stages: storage writing, storage reading and calculation.

[0109] During the write phase, both input signals IN and INb are set high, keeping the QB node consistently high. Data to be written is input from the BL terminal. When the Write Word Line (WWL) is high, M1 is turned on, and the input data is written to the Q point, stored using the parasitic capacitance of the Q point. When WWL is low, the write operation ends.

[0110] During the storage and retrieval phase, the input signal IN is set low and INb is set high, enabling M2 and M3 to perform inverter operations, while M5 is turned off to prevent corruption of the read data. At this time, node QB is the inverted result of node Q. When the Read WordLine (RWL) is high, M4 is turned on, and node QB reads the data stored in node Q into RBL through M4. To read the correct stored data, each column requires an inverter to restore QB to the stored value of Q before outputting it externally. When RWL is low, the read operation ends.

[0111] During the calculation phase, both WWL and RWL are set low. The calculation is divided into two phases: pre-charging and capacitive coupling. The input signals IN and INb are an opposite pair of signals. In the pre-charging phase, regardless of the input activation value, IN is always high and INb is always low. QB is pre-charged to a high level VDD, and CBL is pre-charged to a high level VDD. In the capacitive coupling phase, when the input activation value is 1 (DATA_IN = 1), IN is low, INb is high, M5 is turned off, and QB reads the data from storage cell Q. For the first cell storing a bit "1", QB is pulled low, and this voltage change is coupled to CBL through capacitor CMOM, causing the voltage on CBL to drop by ΔV, indicating that a "1" has been accumulated. For the second cell storing a bit "0", QB remains high, so there is no coupled voltage change on CBL, indicating that a "0" has been accumulated. During the capacitive coupling stage, when the input activation value is 0 (DATA_IN=0), IN is set high, INb is set low, the cell's read path M2-M3 is turned off, M5 is turned on, and QB remains at a high level VDD. At this time, no matter what value is stored in the storage node Q, the voltage at point QB does not change. Therefore, there is no voltage change on CBL, which indicates that when the activation value is 0, "0" is accumulated on CBL.

[0112] Optionally, another embodiment of the present invention provides a 7T2CeDRAM memory sub-unit specifically comprising:

[0113] A sixth transistor, wherein the first terminal of the sixth transistor is connected to a bit line, and the gate of the sixth transistor is connected to a first read word line;

[0114] The seventh transistor has its first terminal connected to the first terminal of the sixth transistor and then connected to the same bit line as the first terminal of the sixth transistor. The gate of the seventh transistor is connected to the second read word line.

[0115] The eighth transistor and the ninth transistor form an inverter, and one end of the inverter is connected to the second end of the sixth transistor and the second end of the seventh transistor.

[0116] The third capacitor is disposed between the second terminal of the sixth transistor and the second terminal of the seventh transistor and the first terminal of the inverter, and is used to store the data received by the third storage node. The third storage node is disposed on the upper plate of the third capacitor and is used to receive the data written by the sixth transistor and the seventh transistor.

[0117] A tenth transistor, the first terminal of which is connected to the second terminal of the inverter, the second terminal of which is connected to the read bit line, the gate of which is connected to the read word line, and a fourth memory node is provided between the tenth transistor and the second terminal of the inverter;

[0118] The eleventh transistor, the first terminal of which is connected to the POS terminal, and the gate of which is connected to the third memory node;

[0119] The twelfth transistor has its first terminal connected to the second terminal of the eleventh transistor, its gate connected to the fourth storage node, the fourth storage node being disposed between the inverter and the first terminal of the tenth transistor, and its second terminal connected to the NEG terminal.

[0120] The fourth capacitor has its upper plate connected to the second terminal of the eleventh transistor and the first terminal of the twelfth transistor, and its lower plate connected to the calculation bit line.

[0121] Furthermore, the sixth transistor, the seventh transistor, and the third capacitor form a writing circuit. The first terminals of the sixth transistor and the seventh transistor receive the data to be written, and the second terminals of the sixth transistor and the seventh transistor output the data to be written and store it in the third storage node on the upper plate of the third capacitor, thus completing the data writing.

[0122] The eighth and ninth transistors form an inverter, and the inverter and the tenth transistor form a read circuit. The inverter inverts the data stored in the third storage node and stores it in the fourth storage node. The tenth transistor reads the data stored in the fourth storage node onto the read bit line to complete the data reading.

[0123] The eleventh transistor, the twelfth transistor, and the fourth capacitor form a computing circuit. The first terminal of the eleventh transistor and the second terminal of the twelfth transistor respectively receive the differential input data from the POS terminal and the NEG terminal. The voltage change of the upper plate of the fourth capacitor is output to the computing bit line. The voltage change of the upper plate of the fourth capacitor is determined by the stored data of the third storage node or the data stored in the fourth storage node, thus completing the data calculation.

[0124] Based on the above embodiments, Figure 5 A structural diagram of the 7T2C storage sub-unit provided in an embodiment of the present invention is shown below. Figure 5As shown, similar in function to the 5T2C structure, this 7T2C memory cell can also be divided into three parts: storage writing, storage reading, and calculation. Transistors M1-M2 and capacitor C1 form the writing module. The data to be written is input from BL, passes through M1, and is stored on the upper plate node Q of C1. Transistors M3-M4, forming an inverter, together with M5, form the read module. The inverted result QB of the data stored at point Q is read out onto RBL through M5. Transistors M6-M7 and capacitor C2 form the calculation module. Input data is differentially input to M6 / M7 through the POS / NEG terminals. The voltage on the upper plate of C2 is changed according to the data Q / QB stored in the cell, thus affecting the voltage change on the Computer Bit Line (CBL). This voltage change ΔV represents the calculation result of the cell.

[0125] In one specific embodiment, taking a 5T2C / 7T2C unit structure as an example of an in-memory computing system that supports fully parallel computation of deep convolution channels, the following can be used: Figure 1 See the overall architecture diagram of the in-memory computing chip, which includes the aforementioned in-memory computing system supporting fully parallel computation of deep convolutional orbits, as follows: Figure 1 As shown, the peripheral circuitry of the in-memory computing chip includes timing control circuitry, word line driver circuitry, and memory read / write interface. The in-memory computing system supporting fully parallel computation of deep convolution channels consists of a read / write separated eDRAM memory sub-cell array, weight configuration circuitry, and ADC quantization circuitry. Each eDRAM memory sub-cell has a 5T2C / 7T2C cell architecture; every eight adjacent 5T2C / 7T2C cells in the column direction represent an 8-bit parameter. This module is a Multi-bit Data Unit (MbDU). One column can represent a feature. Figure 1 There are N 3×3 8-bit weight data points on each input channel. During depthwise convolution computation, each column of the storage sub-cell array can simultaneously compute a matrix multiplication of a 1-bit activation input vector of size (9,1) and an 8-bit weight vector of size (9,N). Every 9 columns of the storage sub-cell array are divided into a channel-wise parallel block (CWPB), which includes N parallel computation modules. Each CWPB can represent N 3×3 8-bit convolution parameters from one input channel. Each CWPB module also has an update circuit for refreshing the data stored within that module.

[0126] For a 3×3 depthwise convolution, each column of the storage sub-cell array can simultaneously compute N 3×3 convolution operations on one input channel. The results of the array computation are first simulated by recombining the 8-bit weights through a weight configuration circuit, representing the simulated result of multiplying and accumulating the 9 1-bit inputs and 8-bit weights in each convolution operation, and then quantized into integer data output by the ADC.

[0127] The in-memory computing system provided by this invention supports fully parallel computation of deep convolution channels, which can improve the computational parallelism of the in-memory computing array and thus improve energy efficiency.

[0128] Based on the in-memory computing system supporting fully parallel computation of depthwise convolution channels provided in the above embodiments, this invention provides an in-memory computing method for depthwise convolution computation in depthwise separable neural networks, which supports fully parallel computation of depthwise convolution channels. This method is implemented using an in-memory computing system that supports fully parallel computation of depthwise convolution channels. Figure 6 This is a flowchart illustrating the in-memory computing method supporting fully parallel computation of depthwise convolution channels provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the method includes:

[0129] Step 110: Obtain the input feature map of each input channel in the deep separable neural network to be calculated by depth convolution. The deep separable neural network includes M input channels, and each input channel is divided into N sub-feature maps of size k×k.

[0130] Step 120: Divide the input feature map into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of first columns and then rows;

[0131] Step 130: Store the N k×k q-bit activation values ​​of an input channel through an N-row parallel computing module;

[0132] Step 140: Perform a multiplication and accumulation operation on the input weight parameters of size (k×k,1) vector and the N k×k q-bit activation value storage data in the column where the input feature map is located, to obtain the N×q multiplication and accumulation operation results output by the input channel;

[0133] Step 150: The N×q multiply-accumulate results output from the input channel are simulated and weighted by q bits using a weight configuration circuit to obtain N multiply-accumulate results.

[0134] Step 160: Quantize and output the N multiplication and accumulation calculation results using an ADC quantization circuit;

[0135] Step 170: Based on the sliding direction of the convolution kernel in the input feature map, the activation value update circuit performs a local cyclic update of the activation value storage data inside the parallel computing module.

[0136] The steps described above will be explained in detail below with reference to specific embodiments.

[0137] Understandably, deep separable networks replace the traditional k×k multiply-accumulate operation of N channels in convolution with a k×k multiply-accumulate operation of depthwise convolution, reducing the accumulation length of the multiply-accumulate operation by a factor of N. Furthermore, because the convolution window needs to move across the input feature map, the activation values, fixed in the in-memory array, require switching and updating operations. This means that traditional in-memory arrays can only perform calculations row by row, resulting in low energy efficiency. In other words, fixing the activation value data of deep separable networks in traditional in-memory arrays suffers from low energy efficiency.

[0138] To address the aforementioned issues, this invention employs a reasonable mapping method to achieve full reuse of weight data within the channel, thereby enhancing the computational parallelism of the in-memory computing array and improving energy efficiency.

[0139] Specifically, Figure 7 The following is a diagram of a computational mapping architecture that supports fully parallel computation of depth-separable network channels, as provided in the embodiments of the present invention. Figure 7 As shown, the input feature map computed by depthwise convolution is divided into multiple non-overlapping 3×3 sub-maps in different input channels, arranged in a column-first, row-second adjacent order. The parallel computing module can store nine 8-bit activation values ​​of a 3×3 sub-map tiled into a (9,1) vector. That is, each row in the CWPB can store nine 8-bit activation values ​​of a 3×3 sub-map tiled into a (9,1) vector, and each CWPB structure can store the activation values ​​of one input channel. For an input weight parameter of size (9,1) vector for a given channel, the activation values ​​stored in the CWPB for that channel can be used to compute the input, thus enabling the computation of all parameters in that channel to be completed with a single input. Compared to previous methods that could only perform partial parameter computation for a single channel, this significantly improves computational throughput.

[0140] Specifically, in traditional methods, when the convolutional kernel slides across the feature map, the data stored in the in-memory array needs to be updated. Updating this stored data consumes a significant amount of power. To conserve this power, more complex selection control logic needs to be introduced into the array. This results in the array only being able to compute the nine parameters of a single 3×3 sub-map in the feature map for a single input, meaning it can only perform parameter calculations for a portion of the channels. This invention employs a reasonable mapping method to achieve parallel computation across all channels.

[0141] Furthermore, the parallel computing process requires updates to the corresponding convolution kernel sliding operation. In other words, the fully parallel computing scheme can be used in conjunction with the update scheme for the corresponding convolution kernel sliding operation.

[0142] On one hand, the core of a deep separable neural network is deep separable convolution, which consists of depthwise convolution and pointwise convolution. During depthwise convolution computation, a k×k kernel is configured for each input channel, and the deep convolutional layer corresponding to M input channels has N k×k convolutions. Because each k×k convolution corresponds to a different activation value of the input feature map, compared to traditional convolutional neural network operations with N input channels, M output channels, and k×k convolutions, the reuse rate of input activation values ​​is reduced by a factor of M. This results in a decrease in throughput by a factor of M when deploying this network in a traditional in-memory computing chip array, without a reduction in power consumption, thus reducing overall energy efficiency by a factor of M. On the other hand, because model weight data is reused more times than input activation values ​​in a deep separable neural network, fixing the input activation values ​​in an in-memory computing array has a greater energy efficiency advantage. This is because the same weight input can perform more computations with the input activation values, achieving higher parallelism, thereby improving throughput and energy efficiency.

[0143] On the other hand, once the input activation values ​​are fixed in the in-memory array, the operation of updating the activation values ​​with the convolution window is inevitable. In traditional designs, data is reloaded from off-chip memory in each computation cycle. This operation consumes a lot of energy, so updating the activation values ​​directly on-chip through simple shift operations is a more efficient approach.

[0144] To address the aforementioned issues, preferably, during the computation phase, an activation value update circuit can be configured for each column of the parallel computing modules. This activation value update circuit is used to implement local cyclic updates of the activation values. For example, the activation value update circuit can be configured as a tri-state inverter, which connects the bit line and read bit line of the column containing the parallel computing module whose activation value needs updating. When the convolution kernel slides vertically in the feature map, it updates the activation values ​​of adjacent k columns every cycle. When the convolution kernel slides horizontally in the feature map, it updates the activation values ​​of every k columns every cycle.

[0145] Specifically, to avoid activating the entire eDRAM array for each weight input, this application proposes a column-level update scheme that slides in both directions to adapt to depthwise convolution calculations. This scheme can achieve data cyclic updates within the array, resulting in 3.5 times lower power consumption compared to updating parameters off-chip. Furthermore, using a partial update method can further reduce power consumption by 2 / 3 for each update. Figure 8 This is a flowchart illustrating the activation value local cyclic update method adapted to a fully parallel computing architecture provided in an embodiment of the present invention, as shown below. Figure 8 As shown, each CWPB updates the activation values ​​of only three columns at a time. After three cycles of update calculations, all convolution calculations for the activation values ​​of one input channel are completed. Specifically, when the convolution kernel slides vertically in the feature map, the activation values ​​of the three adjacent columns are updated each cycle. In the first cycle (cycle 0), the activation values ​​of the first three columns are updated. Within the array, the data of A9-A10 replaces the original data of A0-A3, the data of A18-A20 replaces the original data of A9-A11, and so on. At the same time, the input weights are also cyclically shifted to the right, so that the updated activation values ​​(A9-A11, A3-A4, A5-A6) and the cyclically shifted weights (W6-W8, W0-W2, W3-W5) correspond to the operation of the convolution kernel sliding one step vertically. The activation value update circuit adopts... Figure 6 The circuit shown in the lower right corner only requires a single tri-state inverter connected to the BL and RBL of the same column to achieve column-level data updates. During an update, the RWL of the data source cell is turned on, and the WWL of the target cell is turned on. The data Q from the source cell can then be transmitted to the BL of the same column via the tri-state inverter in the update circuit through the RBL, refreshing the data Q of the target cell, thus achieving cyclic updates within the array.

[0146] As the convolutional kernel slides horizontally across the feature map, the activation values ​​are updated every three columns per cycle. In the first cycle (cycle 0), the activation values ​​of columns 1, 4, and 7 are updated. Similar to the cyclic update in the vertical direction, the activation values ​​of A21, A24, and A27 replace the original activation values ​​of A0, A3, and A6. The weights complete the corresponding cyclic update, so that the activation values ​​(A21, A1, A2) - (A24, A4, A5) - (A27, A7, A8) and the weights (W2, W0, W1) - (W5, W3, W4) - (W8, W6, W7) satisfy the corresponding operation of the convolutional kernel sliding one step horizontally across the feature map.

[0147] In traditional computing architectures, depthwise convolution requires a loop of three processes: multiply-accumulate (MAC), result storage, and convolution window sliding. In the in-memory computation execution sequence, the MAC and convolution window sliding of different input channels are two completely independent parts. Based on this, each input channel operation can be set as an independent execution part. For example, when performing MAC on channel i, the convolution window sliding operation can be performed on channel i+1, or when performing MAC on channel i+1, the convolution window sliding operation can be performed on channel i. Figure 9 This is a schematic diagram of the pipeline timing design for depthwise convolution computation and memory update provided in an embodiment of the present invention, as shown below. Figure 9 As shown, when the eDRAM array is working, two adjacent channels perform calculations in each cycle. One channel performs multiplication and accumulation, while the other channel performs convolution window sliding, thus saving computation time for depthwise convolution. Compared to traditional architectures, this approach... Figure 9 The timing design scheme shown can save 1.6 times the time.

[0148] This invention provides a system that supports fully parallel computation of depthwise convolution channels, increasing the parallelism of array computation and thus improving energy efficiency. Furthermore, it proposes an activation value update scheme adapted to depthwise separable convolutions, achieving efficient updates that match the movement of the convolution window. By reducing the update frequency of array data, it lowers the array update power consumption, thereby improving energy efficiency.

[0149] This invention also provides an in-memory computing chip, including the in-memory computing system that supports fully parallel computation of deep convolution channels as provided in any of the above embodiments.

[0150] As another implementation of the in-memory computing chip provided in the above embodiments, Figure 10 This is a schematic diagram of the structure of the in-memory computing chip provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the in-memory computing chip provided in this embodiment of the invention may include multiple in-memory computing systems that support fully parallel computation of deep convolution channels, thereby enabling parallel computation of more input channels.

[0151] This invention also provides the energy and area distribution of in-memory computing chips when performing depthwise separable convolution calculations, such as... Figure 11 As shown, Figure 11 This diagram illustrates the energy and area distribution of the in-memory computing chip provided in this embodiment of the invention when performing depthwise separable convolution calculations. It primarily includes analog and digital components. The analog component comprises three parts: eDRAM array multiplication-accumulation calculation, ADC quantization, and column-level data update circuitry. The digital component includes the chip's timing control circuitry, array row and column driving and decoding circuitry, and memory read / write I / O interface circuitry. The array multiplication-accumulation and ADC quantization are the most significant components of array power consumption, accounting for 33.9% and 30% of the total power consumption, respectively. In terms of area distribution, the eDRAM array has the largest area, occupying 68.6% of the chip area, followed by the ADC at 18.8%.

[0152] Figure 12 The following are normalized comparison results of update power consumption for different architectures during deep separable neural network computation provided in embodiments of the present invention, such as... Figure 12As shown, achieving high computational energy efficiency in traditional in-memory computing chips requires a storage-based computing scheme that stores activation values. This necessitates updating the entire array's data every time the convolution kernel slides in each cycle. While the ISSCC23 work optimizes this aspect, it is still slightly inferior to the work of this invention. The energy consumption for updating is based on the traditional in-memory computing architecture. Figure 10 The results show that the power consumption for activation value updates in this invention can be reduced by 91.6% compared to the traditional architecture and by 58.2% compared to the ISSCC23, thereby greatly improving the energy efficiency of in-memory computing chips when computing depth-separable convolutions.

[0153] Comparison of this invention with other advanced works Figure 13 As shown, Figure 13 The performance comparison chart of the in-memory computing chip provided in this embodiment of the invention with other advanced works shows that, when using an 8-bit quantized MobileNet-V2 model for inference on the CIFAR10 dataset, it achieves an energy efficiency of 40.2 TOPS / W and an area efficiency of 4.22 TOPS / mm². The energy efficiency of this invention is more than 4.97 times that of similar advanced works, and the area efficiency is more than 1.45 times that of similar advanced works.

[0154] Figure 14 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 14As shown, the electronic device may include: a processor 1410, a communication interface 1420, a memory 1430, and a communication bus 1440, wherein the processor 1410, the communication interface 1420, and the memory 1430 communicate with each other through the communication bus 1440. The processor 1410 can call logical instructions in the memory 1430 to execute an in-memory computing method that supports fully parallel computation of deep convolution channels. The method includes: acquiring the input feature maps of each input channel in a deep separable neural network to be deep convolutionally computed, wherein the deep separable neural network includes M input channels, and each input channel is divided into N k×k sub-feature maps; dividing the input feature maps into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of column-first, row-second; storing the N k×k q-bit activation values ​​of the input channels through an N-row parallel computing module; and (k×k 1) The input weight parameter of the vector size is multiplied and accumulated with the N k×k q-bit activation value storage data in the column of the input feature map to obtain N×q multiply-accumulate results output by the input channel; the N×q multiply-accumulate results output by the input channel are simulated and weighted by q bits by the weight configuration circuit to obtain N multiply-accumulate calculation results; the N multiply-accumulate calculation results are quantized and output by the ADC quantization circuit; according to the sliding direction of the convolution kernel in the input feature map, the local cyclic update of the activation value storage data inside the parallel computing module is performed by the activation value update circuit.

[0155] Furthermore, the logical instructions in the aforementioned memory 1430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the in-memory computing method that supports fully parallel computation of deep convolution channels provided by the above methods. The method includes: acquiring the input feature maps of each input channel in a deep separable neural network to be deep convolutionally computed, wherein the deep separable neural network includes M input channels, and each input channel is divided into N k×k sub-feature maps; dividing the input feature maps into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of column-first, row-second; and storing the sub-maps through an N-row parallel computing module. The system stores N k×k q-bit activation values ​​in the input channel; it performs a multiplication and accumulation operation on the input weight parameters of size (k×k,1) and the N k×k q-bit activation values ​​stored in the column of the input feature map, resulting in N×q multiplication and accumulation results output by the input channel; it then performs q-bit simulated weighting on the N×q multiplication and accumulation results output by the weight configuration circuit, resulting in N multiplication and accumulation calculation results; finally, it quantizes and outputs the N multiplication and accumulation calculation results through the ADC quantization circuit; and finally, it performs local cyclic updates of the activation value storage data within the parallel computing module according to the sliding direction of the convolution kernel in the input feature map through the activation value update circuit.

[0157] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an in-memory computing method that supports fully parallel computation of deep convolution channels, as provided by the methods described above. This method includes: acquiring input feature maps of each input channel in a deep separable neural network to be deep convolutionally computed, wherein the deep separable neural network includes M input channels, and each input channel is divided into N k×k sub-feature maps; dividing the input feature maps into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of column-first, row-second; and storing the N k×k sub-maps of the input channels through an N-row parallel computing module. The system stores q-bit activation values; it performs multiplication and accumulation operations on the input weight parameters of size (k×k,1) and the N k×k q-bit activation values ​​stored in the column of the input feature map to obtain N×q multiplication and accumulation results output by the input channel; it then performs q-bit simulated weighting on the N×q multiplication and accumulation results output by the input channel through a weight configuration circuit to obtain N multiplication and accumulation calculation results; finally, it quantizes and outputs the N multiplication and accumulation calculation results through an ADC quantization circuit; and then, based on the sliding direction of the convolution kernel in the input feature map, it performs local cyclic updates of the activation value storage data within the parallel computing module through an activation value update circuit.

[0158] This invention proposes a computing architecture that supports fully parallel computation of depth-separable network channels, which can increase the parallelism of array computation and thus improve energy efficiency; an activation value update scheme adapted to depth convolution computation, which realizes efficient updates matching the movement of the convolution window, and reduces the power consumption of array updates by reducing the update frequency of array data, thereby improving energy efficiency; and a pipeline timing design for depth convolution computation and memory updates, which designs the convolution multiplication-accumulation MAC computation and the movement operation of the convolution sliding window in a pipeline form, thereby improving the system throughput.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An in-memory computing system supporting deep convolutional channel full parallel computation, comprising: The system is used for deep separable neural networks, and the system includes: The parallel computing module includes k×k multi-bit data units, each of which includes q storage sub-units. Each storage sub-unit is used for writing, reading, and calculating 1-bit stored data, while the multi-bit data units are used for synchronous writing, reading, and calculating q-bit stored data. The parallel computing module stores k×k q-bit activation value storage data and performs simulated multiplication and accumulation operations with input weight parameters of size (k×k,1), where k and q are both positive integers greater than or equal to 2. A weight configuration circuit, connected to the parallel computing module, is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module, to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data. An ADC quantization circuit, connected to the weight configuration circuit, is used to quantize the multiply-accumulate simulation result to obtain an integer data output. An activation value update circuit is provided, which connects the bit line and the read bit line of the column where the parallel computing module is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

2. A memory computing system supporting fully parallel computation of depthwise convolution channels, characterized in that, The system includes: There are M×N parallel computing modules, each of which includes k×k multi-bit data units, and each multi-bit data unit includes q storage sub-units. The storage sub-units are used for writing, reading, and calculating 1-bit stored data, and each multi-bit data unit is used for synchronous writing, reading, and calculating q-bit stored data. The parallel computing modules are used to store q-bit activation value storage data of size k×k, and to perform simulated multiplication and accumulation operations on the q-bit activation value storage data of size k×k and input weight parameters of size (k×k,1), where M and N are positive integers greater than or equal to 1, and k and q are positive integers greater than or equal to 2. A weight configuration circuit, connected to the parallel computing module, is used to perform bit-by-bit simulation recombination of the simulated multiply-accumulate operation result output by the parallel computing module, to obtain the simulated multiply-accumulate result of 1 bit input weight parameter and q bit activation value storage data. An ADC quantization circuit, connected to the weight configuration circuit, is used to quantize the multiply-accumulate simulation result to obtain an integer data output. An activation value update circuit is provided, which connects the bit line and the read bit line of the column where the parallel computing module is located. The activation value update circuit is used to perform local cyclic updates of the activation value storage data inside the parallel computing module.

3. The in-memory computing system supporting fully parallel computation of depthwise convolution channels according to claim 1 or 2, characterized in that, The storage sub-units are connected to the same computation bit line.

4. The in-memory computing system supporting fully parallel computation of depthwise convolution channels according to claim 1 or 2, characterized in that, The storage sub-unit is an embedded dynamic random access memory (eDRAM) unit.

5. The in-memory computing system supporting fully parallel computation of depthwise convolution channels according to claim 4, characterized in that, The eDRAM unit includes: A first transistor, wherein a first terminal of the first transistor is connected to a bit line, and a gate of the first transistor is connected to a write word line; The second transistor has its gate connected to the second terminal of the first transistor, and the second terminal of the second transistor is connected to a power supply. A third transistor, wherein the gate of the third transistor is connected to the second terminal of the first transistor, the first terminal of the third transistor is connected to the first input signal, and the second terminal of the third transistor is connected to the first terminal of the second transistor; A fourth transistor, wherein the first terminal of the fourth transistor is connected to the first terminal of the second transistor and the second terminal of the third transistor; the second terminal of the fourth transistor is connected to the read word line; and a second memory node is provided between the first terminal of the fourth transistor and the first terminal of the second transistor and the second terminal of the third transistor. The fifth transistor has a first terminal connected to a power supply, a second terminal connected to the second terminal of the third transistor, and a gate connected to a second input signal, wherein the second input signal is the opposite of the first input signal. A first capacitor is disposed between the connection line between the gate of the third transistor and the gate of the second transistor and the second terminal of the first transistor, and is used to store data received by a first storage node. The first storage node is disposed on the upper plate of the first capacitor and is used to receive data written by the first transistor. The second capacitor is disposed between the second terminal of the fifth transistor and the calculation bit line, and is used to output the voltage change on the upper plate of the second capacitor to the calculation bit line. The voltage change on the upper plate is obtained according to the data input by the fifth transistor or the data stored in the first storage node.

6. The in-memory computing system supporting fully parallel computation of depthwise convolution channels according to claim 4, characterized in that, The eDRAM unit includes: A sixth transistor, wherein the first terminal of the sixth transistor is connected to a bit line, and the gate of the sixth transistor is connected to a first read word line; The seventh transistor has its first terminal connected to the first terminal of the sixth transistor and then connected to the same bit line as the first terminal of the sixth transistor. The gate of the seventh transistor is connected to the second read word line. The eighth transistor and the ninth transistor form an inverter, and one end of the inverter is connected to the second end of the sixth transistor and the second end of the seventh transistor. The third capacitor is disposed between the second terminal of the sixth transistor and the second terminal of the seventh transistor and the first terminal of the inverter, and is used to store the data received by the third storage node. The third storage node is disposed on the upper plate of the third capacitor and is used to receive the data written by the sixth transistor and the seventh transistor. A tenth transistor, the first terminal of which is connected to the second terminal of the inverter, the second terminal of which is connected to the read bit line, the gate of which is connected to the read word line, and a fourth memory node is provided between the tenth transistor and the second terminal of the inverter; The eleventh transistor, the first terminal of which is connected to the POS terminal, and the gate of which is connected to the third memory node; The twelfth transistor has its first terminal connected to the second terminal of the eleventh transistor, its gate connected to the fourth storage node, the fourth storage node being disposed between the inverter and the first terminal of the tenth transistor, and its second terminal connected to the NEG terminal. The fourth capacitor has its upper plate connected to the second terminal of the eleventh transistor and the first terminal of the twelfth transistor, and its lower plate connected to the calculation bit line.

7. The in-memory computing system supporting fully parallel computation of depthwise convolution channels according to claim 1 or 2, characterized in that, The activation value update circuit is configured as a tri-state inverter, which is connected to the bit line and read bit line of the column where the parallel computing module is located.

8. A storage-based computing method supporting fully parallel computation of depthwise convolution channels, characterized in that, The method is used for depthwise convolution computation in depthwise separable neural networks. The method is implemented using the in-memory computing system supporting fully parallel computation of depthwise convolution channels as described in any one of claims 1-7. The method includes: To obtain the input feature maps of each input channel in a deep separable neural network to be calculated by depthwise convolution, the deep separable neural network includes M input channels, and each input channel is divided into N sub-feature maps of size k×k; The input feature map is divided into M columns and N rows of non-overlapping k×k sub-maps in an adjacent order of first columns and then rows; The N-row parallel computing module stores the N k×k q-bit activation values ​​in the input channel. The input weight parameters of size (k×k,1) are multiplied and accumulated with the N k×k q-bit activation values ​​stored in the column of the input feature map to obtain the N×q multiply-accumulate results output by the input channel. The N×q multiply-accumulate results output from the input channel are simulated and weighted by q bits using a weight configuration circuit to obtain N multiply-accumulate results. The N multiplication and accumulation calculation results are quantized and output using an ADC quantization circuit. Based on the sliding direction of the convolution kernel in the input feature map, the activation value update circuit performs a local cyclic update of the activation value storage data inside the parallel computing module.

9. The in-memory computing method supporting fully parallel computation of depthwise convolution channels according to claim 8, characterized in that, The step of performing a local cyclic update of the activation value storage data within the parallel computing module based on the sliding direction of the convolution kernel in the input feature map, through the activation value update circuit, specifically includes: As the convolutional kernel slides vertically in the input feature map, the activation value update circuit updates the activation values ​​of adjacent k columns in each cycle. As the convolutional kernel slides horizontally across the input feature map, the activation value update circuit updates the activation values ​​every k columns in each cycle.

10. The in-memory computing method supporting fully parallel computation of depthwise convolution channels according to claim 8, characterized in that, The method further includes: When the i-th input channel performs multiplication-accumulation calculation, the (i+1)-th input channel performs a convolution kernel sliding operation; or, When the (i+1)th input channel performs multiply-accumulate calculation, the ith input channel performs a convolution kernel sliding operation; Where 1≤i≤M.

11. A memory computing chip, characterized in that, The in-memory computing chip includes the in-memory computing system that supports fully parallel computation of deep convolution channels as described in any one of claims 1-7.

Images