Storage and computing integrated module, chip and electronic device based on input data sparsity

Abstract

The present application discloses a storage-computation integrated module, chip and electronic device based on the sparsity of input data, in which a weight parameter storage array is used to store weight parameters; a bit multiplier is used to receive a weight read enable signal and input feature data, and when the weight read enable signal is enabled, a weight parameter in the weight parameter storage array is selected according to the weight read enable signal, and the bit of the input feature data is multiplied by the selected weight parameter; a storage readout circuit is used to read the product to a logic operation unit when the bit of the input feature data is 1, and does not perform a readout operation when the bit of the input feature data is 0; and a logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize multiplication and accumulation of the input feature data and the weight parameter. The present application can reduce the power consumption of a large number of concurrent data reads, and reduce the requirements for data output and data transmission bandwidth.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a storage and computing integrated module, chip and electronic device based on the sparsity of input data. Background Art

[0002] With the development of the artificial intelligence industry, AI applications have continuously increased the computing power and storage requirements of chips, mainly in terms of high computing power, high storage capacity, and high data transmission bandwidth. Storage-computing integrated technology helps solve the "storage wall" and "power wall" problems under the traditional von Neumann architecture. The von Neumann architecture requires data to be continuously "read and written" between the memory unit and the processing unit, so the data transmission back and forth between the two consumes a lot of transmission power.

[0003] The "storage wall" refers to the fact that the performance of the memory cannot keep up with the performance of the CPU (central processing unit), causing the CPU to spend a lot of time waiting for the memory to complete read and write operations, thereby reducing the overall performance of the system. The "storage wall" has become a major obstacle to data computing applications. In particular, the biggest challenge of deep learning acceleration is the frequent movement of data between computing units and storage units.

[0004] The advantage of integrated storage and computing is that it breaks the storage wall. It performs calculations within the storage unit, eliminating unnecessary data movement delays and power consumption, using storage units to improve computing power, improving computing efficiency by hundreds or thousands of times, and reducing costs.

[0005] Sparsity affects the computational efficiency of machine learning because sparse matrices contain a large number of zero values, resulting in a waste of resources, an increase in unnecessary computation, and low storage space utilization. The most critical issue is the waste of resources, because during the training process of machine learning, the algorithm often needs to traverse the entire data set. If the data set is sparse, that is, it contains a large number of zeros or meaningless information, this causes the algorithm to process a large amount of data that will not affect the results. This not only makes the processing time longer, but also consumes more computing resources, resulting in low efficiency and high power consumption. Summary of the invention

[0006] In view of this, the present application provides a storage and computing integrated module, chip and electronic device based on the sparsity of input data to solve some or all of the above-mentioned problems that exist when facing data sparsity.

[0007] The present application provides a storage-computation integrated module based on input data sparsity, the storage-computation integrated module comprising: a weight parameter storage array, a bit multiplier, a storage readout circuit and a logic operation unit;

[0008] The weight parameter storage array is used to store weight parameters;

[0009] The bit multiplier is used to receive a weight read enable signal and input feature data, and when the weight read enable signal is enabled, select a weight parameter in the weight parameter storage array according to the weight read enable signal, and multiply the bit of the input feature data by the selected weight parameter;

[0010] The storage readout circuit is used for reading the product of the multiplication to the logic operation unit when the bit position of the input feature data is 1, and not performing the readout operation when the bit position of the input feature data is 0;

[0011] The logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit position of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight parameter.

[0012] Optionally, the storage read-out circuit includes read-out branches respectively arranged on K rows of bit lines; the read-out branches are used to read the multiplied product to the logic operation unit when the weight read enable signal is enabled and the bit position of the input characteristic data is 1; each of the read-out branches includes a switch unit and a read-out unit electrically connected to the switch unit; the switch unit is used to output a high-level signal to the bit line where it is located when the weight read enable signal and the bit position of the input characteristic data are both 1; the read-out unit is used to read the electrical signal representing the corresponding product through the bit line where it is located when a high-level signal is provided to the bit line where it is located; the switch unit outputs a low-level signal to the bit line where it is located when at least one of the weight read enable signal and the bit position of the input characteristic data is 0, so that the storage read-out circuit does not perform a read-out operation.

[0013] Optionally, each of the switching units includes a PMOS tube and an NMOS tube; the gate of the PMOS tube is connected to the first control signal, the source and the drain are connected between the bit lines of the corresponding row, and the drain is also connected to the drain of the NMOS tube; the gate of the NMOS tube is connected to the first control signal, and the source is grounded; the first control signal is determined according to the weight read enable signal and the input feature data.

[0014] Optionally, the storage readout circuit also includes a first AND gate; one input end of the first AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output a first control signal.

[0015] Optionally, the weight parameter storage array includes K rows and N columns of parameter storage units; the K rows of parameter storage units are correspondingly arranged on K rows of bit lines; the weight read enable signal is used to enable a column of parameter storage units in the N columns of parameter storage units, so that the parameter storage unit outputs an electrical signal representing the weight parameter to the corresponding bit line.

[0016] Optionally, the parameter storage unit includes a parameter storage area and a switch tube; the parameter storage area is used to store an electrical signal characterizing the weight parameter; the switch tube is used to be turned on when the second control signal is 1 to output the electrical signal characterizing the weight parameter to the corresponding bit line, and to be turned off when the second control signal is 0; the second control signal is determined according to the weight read enable signal and the input characteristic data.

[0017] Optionally, the bit multiplier includes N second AND gates respectively connected to the weight parameter storage array; one input end of the second AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output a second control signal.

[0018] Optionally, the logic operation unit is further used to access the input feature data and identify whether a bit of the input feature data is 0, and when the bit of the input feature data is 0, the accumulation result of the logic operation unit remains unchanged.

[0019] The present application also provides a chip, comprising any of the above-mentioned storage and computing modules based on the sparsity of input data.

[0020] The present application also provides an electronic device, comprising any of the above-mentioned storage and computing integrated modules based on the sparsity of input data or any of the above-mentioned chips.

[0021] In the storage-computation integrated module, chip and electronic device based on the sparsity of input data described above, the storage readout circuit reads the weight parameter to the logic operation unit when the bit of the input feature data is 1, and does not perform the readout operation when the bit of the input feature data is 0, so that the storage readout circuit does not generate power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit; the output data of the logic operation unit is the accumulation result, and there is no need to transmit the intermediate data in the convolution operation process, and the data bit width of its output data is reduced. If a higher accumulation operation is implemented in the logic operation unit, the output data bit width can be further compressed. It can be seen that the present application can reduce the power consumption of a large number of concurrent data reads and reduce the requirements for data output and data transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1This is a schematic diagram of the storage-computing integrated module structure of an embodiment of the present application;

[0024] Figure 2 This is a schematic diagram of the storage-computing integrated module structure of another embodiment of the present application;

[0025] Figure 3 This is a schematic diagram of the storage-computing integrated module structure of another embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the weight parameter storage array structure of an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of the weight parameter storage array structure of another embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the structure of a storage readout circuit according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of the storage readout circuit structure of another embodiment of the present application;

[0030] Figure 8 This is a schematic diagram of the storage-computing integrated module structure of another embodiment of the present application;

[0031] Fig. 9 It is a schematic diagram of the storage and computing integrated module structure of another embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application. In the absence of conflict, the following embodiments and their technical features can be combined with each other.

[0033] In order to solve the problem that sparse data sets consume more computing resources and lead to low computing efficiency, some algorithms mainly optimize the data of the weight matrix based on sparsity to reduce unnecessary calculations. However, it is difficult to optimize the sparsity of input data, mainly because of the uncertainty of input data, which needs to be processed in real time inside the chip. Data sparsity varies greatly at different granularities. The smaller the granularity, the higher the sparsity. For example, the probability that an N-bit input vector is all 0 is much lower than the probability that some bits in an N-dimensional input vector are 0. The bit-level sparsity of data is very high. However, the smaller the granularity, the more difficult it is to use data sparsity, which is mainly reflected in the need to perform real-time calculations and judgments on each bit of input data.

[0034] Based on the above problems, the first aspect of the present application provides a storage and computing module based on the sparsity of input data. Figure 1 As shown, the storage-computation integrated module includes a weight parameter storage array 10, a bit multiplier 12, a storage readout circuit 21 and a logic operation unit 22. One input end of the bit multiplier 12 is connected to a weight read enable signal, and the other input end is connected to input feature data. The weight parameter acquisition end is connected to the weight parameter storage array 10; the input end of the storage readout circuit 21 is connected to the weight parameter storage array 10, and the output end is connected to the logic operation unit 22; the output end of the logic operation unit 22 is used to output the convolution operation result.

[0035] The weight parameter storage array 10 is used to store weight parameters. Figure 2 The storage and computing module shown may also include a storage writing circuit 11; the storage writing circuit 11 is used to write weight parameters to the weight parameter storage array 10, for example, Figure 2 As shown, the storage write circuit 11 can access the write data address and weight write data signals, and write the weight write data into the weight parameter storage array 10 according to the write data address.

[0036] The bit multiplier 12 is used to receive a weight read enable signal and input feature data (in a convolutional neural network, it is usually necessary to process the input feature map). When the weight read enable signal is enabled, the weight parameter in the weight parameter storage array is selected according to the weight read enable signal, and the bit of the input feature data is multiplied by the selected weight parameter. The input feature data is used to represent the data output by the previous layer of operation during the convolution operation, and may include at least one bit, wherein the bit value is 1 or 0; for the input feature data, the bit value is 1 to represent that the input feature data is 1, and the bit value is 0 to represent that the input feature data is 0.

[0037] Specifically, when the bit position of the input feature data is 1, the product of the bit position of the input feature data and the corresponding weight parameter weight is the corresponding weight parameter. At this time, the storage read-out circuit 21 reads the corresponding weight parameter from the weight parameter storage array 10 to realize the corresponding product read-out; when the bit position of the input feature data is 0, the storage read-out circuit 21 does not perform the read-out operation at this time, so the storage read-out circuit 21 can directly perform the read-out operation on the weight parameter storage array 10 to realize the corresponding product read-out.

[0038] The storage readout circuit 21 is used to read the multiplied product to the logic operation unit 22 when the bit of the input feature data is 1, and not perform the readout operation when the bit of the input feature data is 0, so that the storage readout circuit 21 does not generate power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit 21. Specifically, the storage readout circuit 21 can read the corresponding weight parameter to the logic operation unit 22 when the bit of the input feature data is 1. Specifically, the storage medium of the storage readout circuit 21 includes any one of SRAM, DRAM and RRAM, and the storage medium can be a volatile storage medium or a non-volatile storage medium.

[0039] The logic operation unit 22 is used to accumulate and store the product read out by the readout circuit 21 when the bit of the input feature data is 1, so as to realize the multiplication and accumulation of the input feature data and the weight parameter. The above-mentioned logic operation unit 22 only needs to output the final convolution operation result, and does not need to transmit the intermediate data in the convolution operation process, which can reduce the requirements for data output and data transmission bandwidth.

[0040] In some embodiments, the weight parameter storage array 10 includes K rows and N columns of parameter storage units. Each parameter storage unit is used to store a weight parameter or an electrical signal representing a corresponding weight parameter, such as Figure 3 As shown, the parameter storage unit in the hth row and jth column stores the weight parameter W(h, j), 1≤h≤K, 1≤j≤N, K is the number of rows of the weight parameter, and N is the number of columns of the weight parameter. The parameter storage units in the Kth row are correspondingly arranged on the bit lines in the Kth row; the bit line in the hth row is denoted as bitline,h.

[0041] The above-mentioned weight read enable signal is used to enable a column of parameter storage units in N columns of parameter storage units, that is, one bit of weight read enable signal corresponds to one column of parameter storage units, such as the weight read enable signal corresponding to the j-th column parameter storage unit can be recorded as wordline,j. When the corresponding weight read enable signal wordline,j of the j-th column parameter storage unit is 1, the electrical signal representing the weight parameter is output to the corresponding bit line, so that the storage read-out circuit 21 can read the corresponding weight signal through each row of bit lines.

[0042] In some examples, reference Figure 4 and Figure 5As shown, each parameter storage unit includes a parameter storage area 102 and a switch tube 101; the first end of the switch tube 101 is connected to the corresponding bit line, the second end is connected to the corresponding storage area 102, and the control end is used to access the corresponding control signal. The parameter storage area 102 is used to store an electrical signal representing the corresponding weight parameter; the switch tube 101 is used to turn on when the accessed control signal (such as the second control signal) is 1, so as to output the electrical signal representing the corresponding weight parameter to the corresponding bit line, so that the storage read-out circuit 21 can obtain the electrical signal representing the weight parameter on the bit line through the bit line, thereby obtaining the corresponding weight parameter; the switch tube 101 is also used to turn off when the accessed control signal is 0, at which time the electrical signal stored in the parameter storage area 102 will not be transmitted to the corresponding bit line. The control signal accessed by the switch tube 101, such as the second control signal, is determined according to the weight reading enable signal and the input characteristic data.

[0043] Optionally, the switch tube 101 is implemented by a transistor such as an NMOS tube, and the parameter storage area 102 is implemented by components such as a capacitor, for example, Figure 4 and Figure 5 As shown, the switch tube 101 includes an NMOS tube, and the parameter storage area 102 includes a capacitor; the gate of the switch tube 101 serves as the control terminal of the switch tube 101 for accessing the control signal, the source is grounded through the capacitor, and the drain is connected to the corresponding bit line.

[0044] Specifically, the second control signal connected to the switch tube 101 in the parameter storage unit can also be determined according to the data reading method of the storage readout circuit 21. In some examples, the storage readout circuit 21 reads the parameter storage units on the odd-numbered bit lines and the even-numbered bit lines in the same way. In this case, the control signals connected to the parameter storage units on the odd-numbered bit lines and the even-numbered bit lines are the same, such as Figure 4 As shown, the second control signal can be determined by performing an AND operation on the weight read enable signal and the input characteristic data, that is, the second control signal takes 1 when the weight read enable signal and the input characteristic data are both 1, and takes 0 when at least one of the weight read enable signal and the input characteristic data takes 0. In other examples, the storage readout circuit 21 performs differential reading on the parameter storage units on the odd-numbered row bit lines and the even-numbered row bit lines. At this time, the control signals connected to the parameter storage units on the odd-numbered row bit lines and the even-numbered row bit lines are different (the control signals connected to the parameter storage units on the odd-numbered row bit lines and the even-numbered row bit lines can be opposite), such as Figure 5 As shown, the second control signal connected to the parameter storage unit on the odd-numbered bit lines can perform an AND operation on the weight read enable signal and the input characteristic data, and the control signal connected to the parameter storage unit on the even-numbered bit lines is opposite to the second control signal connected to the parameter storage unit on the odd-numbered bit lines.

[0045] Specifically, the bit multiplier 12 includes N second AND gates 121 respectively connected to the weight parameter storage arrays, such as Figure 4 , Figure 5 As shown, Figure 4 , Figure 5 Only the second AND gate 121 of the Nth column is numbered, and each column parameter storage unit corresponds to a second AND gate 121. One input end of the second AND gate 121 is used to access the input feature data, the other input end is used to access the weight reading enable signal, and the output end is connected to the control end of the switch tube 101 in the corresponding column, and is used to output the second control signal; taking the second AND gate 121 of the jth column as an example for specific description, on the jth column, one input end of the second AND gate 121 is used to access the input feature data fmi,j of the jth column, the other input end is used to access the weight reading enable signal wordline,j of the jth column, and the output end is connected to the control end of the corresponding switch tube 101 in the jth column.

[0046] In some embodiments, Figure 2 As shown, the storage readout circuit 21 can also access the input feature data so that when the input feature data is 1, the corresponding weight parameter is read out to the logic operation unit 22, and when the input feature data is 0, no readout operation is performed.

[0047] Specifically, refer to Figure 6 As shown, the storage readout circuit 21 includes readout branches respectively arranged on the K rows of bit lines; that is, the hth row of bit lines is provided with a readout branch h, and the readout branch h includes a control terminal, and the control terminal is used to access the corresponding control signal, and the control signal is determined according to the weight read enable signal wordline,j and the input feature data fmi,j. Each readout branch is used to access the weight read enable signal and the input feature data, and when the weight read enable signal is enabled and the bit of the input feature data is 1, the multiplied product is read out to the logic operation unit 22. Specifically, each readout branch can read a column of weight parameters enabled by the weight read enable signal from the weight parameter storage array 10 when the weight read enable signal is enabled and the bit of the input feature data is 1. Taking the example of each read-out branch accessing the weight read enable signal wordline,j and the input feature data fmi,j corresponding to the j-th column, the working process of the read-out branch is explained. If the weight read enable signal wordline,j and the input feature data fmi,j are both 1, each read-out branch reads the corresponding weight parameter from the parameter storage unit on the j-th column in the weight parameter storage array; if at least one of the weight read enable signal wordline,j and the input feature data fmi,j is 0, each read-out branch does not currently perform a read operation.

[0048] In some examples, reference Figure 7As shown, each readout branch includes a switch unit 211 and a readout unit 212. The switch unit 211 includes a control terminal, which is used to access a corresponding control signal (such as a first control signal). When the accessed control signal is 1, the switch unit 211 outputs a high-level signal to the corresponding bit line, and when the accessed control signal is 0, the switch unit 211 outputs a low-level signal to the corresponding bit line; wherein the control signal is determined according to the weighted read enable signal and the input feature data.

[0049] Specifically, the switch unit 211 is used to output a high level signal to the bit line when the weight read enable signal and the bit of the input characteristic data are both 1; the readout unit 212 is used to obtain an electrical signal representing the corresponding product through the bit line when the bit line provides a high level signal to obtain the corresponding product. When at least one of the weight read enable signal and the bit of the input characteristic data is 0, the switch unit 211 outputs a low level signal to the bit line so that the storage readout circuit 21 does not perform a read operation.

[0050] Taking reading the weight parameter at the parameter storage unit of the j-th column as an example, the working process of the above-mentioned switch unit 211 and the read-out unit 212 is explained. If the control end of the switch unit 211 is connected to the weight read enable signal wordline,j and the input characteristic data fmi,j, when the weight read enable signal wordline,j and the input characteristic data fmi,j are both 1, in the parameter storage unit on the j-th column, the switch tube 101 is turned on, and the electrical signal stored in the parameter storage area 102 is transmitted to the corresponding bit line, and the switch unit 211 outputs a high-level signal to the bit line where it is located. At this time, the read-out unit 212 can obtain the electrical signal representing the weight parameter at the parameter storage unit of the j-th column through the bit line where it is located when the bit line where it is located provides a high-level signal, so as to obtain the weight parameter of the j-th column; when at least one of the weight read enable signal wordline,j and the input characteristic data fmi,j is 0, the switch unit 211 outputs a low-level signal to the bit line where it is located, and the read-out unit 212 does not perform read and write operations. The weight read enable signal wordline,j enables the weight parameter read operation stored in the jth column parameter storage unit, and multiple consecutive columns of weight read enable signals can enable the weight parameter read operation stored in multiple consecutive columns of parameter storage units. For example, the weight read enable signals wordline,1~N in the 1st to Nth columns can also be called chip select signals CSL.

[0051] Furthermore, the low-level signal output by the switch unit 211 is directly output to the logic operation unit 22 through the corresponding bit line, and the logic operation unit 22 can accumulate 0 when reading the low-level signal transmitted by the corresponding bit line.

[0052] Optionally, each readout unit 212 may also be connected to a write enable signal WE and other related control signals (such as Figure 7 As shown in write_data1, write_data2, etc.), where the write enable signal WE can be used to enable the read operation of the read unit 212.

[0053] In some examples, the storage readout circuit 21 includes a DRAM storage array, and the structure of the switch unit 211 can refer to Figure 7 As shown, it includes PMOS tubes and NMOS tubes; wherein the PMOS tubes in the hth row are marked as P,h, and the NMOS tubes in the hth row are marked as N,h, that is, in the hth row, the switch unit 211 includes PMOS tubes P,h and NMOS tubes N,h. In each row, the gate of the PMOS tube serves as the control end of the corresponding switch unit 211, and is connected to the first control signal CIM_CSL, and the source and drain are connected between the bit lines of the corresponding row, that is, the connection of the switch unit 211 on the bit line is disconnected, and one side of the disconnected position is a setting parameter storage unit, which can also be called a storage row, and the other side of the disconnected position is a setting readout branch, the source of the PMOS tube is connected to one end of the disconnected position, and the drain of the PMOS tube is connected to the other end of the disconnected position; the drain of the PMOS tube is also connected to the drain of the NMOS tube; the gate of the NMOS tube is connected to the first control signal CIM_CSL, and the source is grounded. When the first control signal CIM_CSL connected to the control end of the switch unit 211 is 1, the NMOS tube is turned off and the PMOS tube is turned on, so that the storage row and the corresponding bit line can be connected, so that the readout unit 212 obtains the electrical signal representing the corresponding weight parameter from the bit line; when the first control signal CIM_CSL connected to the control end of the switch unit 211 is 0, the PMOS tube is turned off and the NMOS tube is turned on, and the drain of the NMOS tube outputs a low-level signal to the logic operation unit 22 through the bit line of the corresponding row. The first control signal CIM_CSL is determined according to the weight reading enable signal and the input characteristic data.

[0054] Specifically, the first control signal CIM_CSL is the result of the AND operation of the weight read enable signal and the input feature data. Figure 7The storage readout circuit 21 shown also includes a first AND gate 213; one input end of the first AND gate 213 is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is respectively connected to the control end of each switch unit 211, for outputting a first control signal CIM_CSL; for the weight read enable signal wordline,j and the input feature data fmi,j corresponding to the j-th column, when the weight read enable signal wordline,j and the input feature data fmi,j are both 1, the first control signal CIM_CSL is 1; when at least one of the weight read enable signal wordline,j and the input feature data fmi,j is 0, the first control signal CIM_CSL is 0.

[0055] Alternatively, if Figure 7 As shown, the read-out unit 212 may include components such as a read-write MOS tube and an amplifier. The control terminals such as the gate of the read-write MOS tube may also be connected to other control signals such as a write enable signal WE. The read-out unit 212 on each row of bit lines is used to read out the electrical signal representing the weight parameter on the row of bit lines when it is turned on, so as to read out the corresponding product.

[0056] The inventor compared the traditional DRAM storage array with the storage read-out circuit 21 provided in this example and found that in the traditional DRAM storage array, the read-out structure of each row usually includes a PMOS tube in the read-out unit and the switch unit. The read-out branch provided in this example only needs to add an NMOS tube in each row, so that the switch unit 211 can disconnect the storage row of the corresponding row from the bit line when the first control signal CIM_CSL connected to its control end is 0. The bit line can be pulled low by the NMOS tube in the switch unit of each row, so that the storage read-out circuit 21 does not perform the read operation and directly outputs a low-level signal to the logic operation unit 22.

[0057] Specifically, Figure 7 The storage and computing module corresponding to the storage readout circuit 21 shown can refer to Figure 8 As shown, each row of bit lines may be provided with components such as a sense amplifier and / or an equalizer for improving reliability. Accordingly, the above-mentioned storage-computing integrated module may also include a sense amplifier and / or an equalizer provided on each row of bit lines; the input end of the sense amplifier is respectively connected to the output end of the corresponding parameter storage unit through the bit line where it is located, and the output end is connected to the readout branch corresponding to the bit line where it is located through the equalizer; the sense amplifier is used to amplify the electrical signal output from the parameter storage unit to the corresponding bit line; the equalizer is used to equalize the amplified electrical signal; in this way, the electrical signal output to the corresponding readout branch through the bit line can be smoother and more stable, which can improve the reliability of the above-mentioned electrical signal readout process.

[0058] In the working process of the storage-computing module, the weight read enable signal corresponding to columns 1 to N can be recorded as wordline, 1 to N (also called chip select signal CSL), and the input feature data corresponding to columns 1 to N is recorded as fmi, 1 to N. Figure 8 In the storage-computation integrated module shown, the first control signal CIM_CSL is obtained by performing AND operation on fmi,1~N and wordline,1~N; the first control signal CIM_CSL can control the gate transistors P,1~P,K of the bit line (i.e., the PMOS transistors in each switch unit). When fmi (i.e., the input feature data) is 0, wordline (i.e., the weight read enable signal) is 1, at which time P,1~P,K disconnects the storage row of the corresponding row from the bit line, and the first control signal CIM_CSL simultaneously pulls the bit line low through N,1~N,K. Figure 8 The storage-computing module shown avoids the need for the traditional DRAM bit line to complete the read operation in four operations: precharge, access, sense, and restore. At this time, the bit line needs to flip between three states: 0, VCC / 2, and VCC, which is the main source of data reading power consumption and system delay. When fmi and wordline are both 1, P,1~P,K are turned on, N,1~N,K are disconnected, and the read branch works normally, sending the read weight parameters to the logic operation unit 22 to complete the multiplication and accumulation operation. It can be seen that this example only adds one transistor to each row inside the DRAM storage array, and does not add a control signal line, so that the storage-computing solution based on the sparsity of the input data can be completed, which can reduce the data reading power consumption during the convolution operation. Especially for the case of bit-level data sparsity, it can greatly reduce the power consumption of the convolution operation, and can make the bit width of the output data better compressed, greatly reducing the requirements for data output and data transmission bandwidth.

[0059] In some embodiments, reference Fig. 9 As shown, the logic operation unit 22 is also used to access the input feature data and identify whether the bit of the input feature data is 0, and when the bit of the input feature data is 0, the accumulation result of the logic operation unit 22 remains unchanged. In this embodiment, the logic operation unit 22 directly accesses the input feature data, and when the bit of the input feature data is 0, the accumulation result remains unchanged, and there is no need to perform the accumulation operation, which can further save power consumption.

[0060] Specifically, a bit of weight read enable signal and a bit of input feature data correspond to a column of parameter storage units respectively. For the weight read enable signal wordline,j and the input feature data fmi,j corresponding to the jth column, when wordline,j=1 and fmi,j=1, the logic operation unit 22 performs accumulation and other operations on the signal representing the weight parameter on the bit line, and the convolution operation sum obtained is: ,in, represents the cumulative sum, represents the input feature data corresponding to the jth column, Indicates the weight parameter stored in the parameter storage unit of the hth row and the jth column, 1≤h≤K, 1≤j≤N. When fmi,j=0, regardless of the value of the weight parameter, the product of the corresponding weight parameter and fmi,j must be 0. At this time, the accumulation result of the logic operation unit 22 remains unchanged, and the weight parameter reading operation of the jth column can be shielded by fmi,j=0, and the voltage of the bit line does not flip. The read power consumption of the storage read circuit 21 is mainly generated when the bit line flips, so when fmi,j=0, there is no data read power consumption. In addition, the logic operation unit 22 directly outputs the accumulation result without transmitting the intermediate data in the convolution operation process. Its output data will also be reduced from the original K-bit data width to the accumulated data width log2(K). If a higher accumulation operation is implemented in the logic operation unit 22, the output data width can be further compressed.

[0061] In the above storage-computation integrated module, when the bit of the input feature data is 1, the storage readout circuit 21 reads the weight parameter to the logic operation unit 22, and when the bit of the input feature data is 0, the readout operation is not performed, so that the storage readout circuit 21 does not generate power consumption when the bit of the input feature data is 0, thereby reducing the power consumption of the storage readout circuit 21; the output data of the logic operation unit 22 is the accumulation result, and there is no need to transmit the intermediate data in the convolution operation process, and the data bit width of its output data is reduced. If a higher accumulation operation is implemented in the logic operation unit 22, the output data bit width can be further compressed. It can be seen that the present application can reduce the power consumption of a large number of concurrent data readings and reduce the requirements for data output and data transmission bandwidth.

[0062] A second aspect of the present application provides a chip, which includes the storage and computing integrated module based on input data sparsity as described in any of the above embodiments, which may include AI computing chips, wireless communication chips, wired communication chips, digital signal processing chips, mobile phone main control chips, AR / VR main control chips and other chips that need to perform convolution operations.

[0063] The above-mentioned chip includes the storage-computing integrated module described in any of the above-mentioned embodiments, and has all the beneficial effects of the storage-computing integrated module based on input data sparsity described in any of the above-mentioned embodiments, which will not be repeated here.

[0064] A third aspect of the present application provides an electronic device, which includes the storage-computing-in-one module based on input data sparsity as described in any of the above embodiments or the chip as described in any of the above embodiments. The chip has all the beneficial effects of the storage-computing-in-one module based on input data sparsity as described in any of the above embodiments, which will not be repeated here.

[0065] Although the present application has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on reading and understanding this specification and the accompanying drawings. The present application includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components, the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (e.g., it is functionally equivalent), even if it is not structurally equivalent to the disclosed structure that performs the function in the exemplary implementation of the present specification shown herein.

[0066] That is, the above description is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, such as the mutual combination of technical features between the embodiments, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

[0067] In addition, in the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, for structural elements with the same or similar characteristics, the present application may use the same or different reference numerals for identification. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0068] In this application, the word "exemplary" is used to mean "used as an example, illustration or description". Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or more advantageous than other embodiments. In order to enable any technician in the field to implement and use the present application, the present application provides the above description. In the above description, various details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other embodiments, well-known structures and processes will not be elaborated in detail to avoid unnecessary details that make the description of the present application obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in the present application.

Claims

1. A storage and computing integrated module based on input data sparsity, characterized in that: The storage-computation integrated module includes: a weight parameter storage array, a bit multiplier, a storage readout circuit and a logic operation unit; The weight parameter storage array is used to store weight parameters; The bit multiplier is used to receive a weight read enable signal and input feature data, and when the weight read enable signal is enabled, select a weight parameter in the weight parameter storage array according to the weight read enable signal, and multiply the bit of the input feature data by the selected weight parameter; The storage readout circuit is used for reading the product of the multiplication to the logic operation unit when the bit position of the input feature data is 1, and not performing the readout operation when the bit position of the input feature data is 0; The logic operation unit is used to accumulate the product read out by the storage readout circuit when the bit of the input feature data is 1, so as to realize multiplication and accumulation of the input feature data and the weight parameter; The storage readout circuit includes readout branches respectively arranged on K rows of bit lines; The readout branch is used to read the multiplied product to the logic operation unit when the weight read enable signal is enabled and the bit of the input feature data is 1; each of the readout branches includes a switch unit and a readout unit electrically connected to the switch unit; The switch unit is used to output a high level signal to the bit line when the weight read enable signal and the bit of the input feature data are both 1; The read-out unit is used to read the electrical signal representing the corresponding product through the bit line when the bit line provides a high-level signal; the switch unit outputs a low-level signal to the bit line when at least one of the bits of the weight read enable signal and the input characteristic data is 0, so that the storage read-out circuit does not perform the read operation.

2. The storage and computing integrated module according to claim 1, characterized in that: Each of the switch units includes a PMOS tube and an NMOS tube; The gate of the PMOS tube is connected to the first control signal, the source and the drain are connected between the bit lines of the corresponding rows, and the drain is also connected to the drain of the NMOS tube; the gate of the NMOS tube is connected to the first control signal, and the source is grounded; the first control signal is determined according to the weight read enable signal and the input characteristic data.

3. The storage and computing integrated module according to claim 1, characterized in that: The storage readout circuit also includes a first AND gate; one input end of the first AND gate is used to access the input feature data, the other input end is used to access the weight read enable signal, and the output end is used to output a first control signal.

4. The storage and computing integrated module according to claim 1, characterized in that: The weight parameter storage array includes K rows and N columns of parameter storage units; the K rows of parameter storage units are correspondingly arranged on K rows of bit lines; the weight read enable signal is used to enable a column of parameter storage units in the N columns of parameter storage units, so that the parameter storage unit outputs an electrical signal representing the weight parameter to the corresponding bit line.

5. The storage and computing integrated module according to claim 4, characterized in that: The parameter storage unit includes a parameter storage area and a switch tube; The parameter storage area is used to store the electrical signal representing the weight parameter; The switch tube is used to be turned on when the second control signal is 1 to output the electrical signal representing the weight parameter to the corresponding bit line, and turned off when the second control signal is 0; The second control signal is determined according to the weight reading enable signal and the input feature data.

6. The storage and computing integrated module according to claim 1, characterized in that: The bit multiplier includes N second AND gates respectively connected to the weight parameter storage array; One input terminal of the second AND gate is used to access the input feature data, another input terminal is used to access the weight read enable signal, and an output terminal is used to output a second control signal.

7. The storage and computing integrated module according to claim 1, characterized in that: The logic operation unit is also used to access the input feature data and identify whether the bit of the input feature data is 0, and when the bit of the input feature data is 0, the accumulation result of the logic operation unit remains unchanged.

8. A chip, characterized in that: It includes the storage and computing integrated module based on input data sparsity as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: It comprises the storage-computing integrated module based on input data sparsity as described in any one of claims 1 to 7 or the chip as described in any one of claim 8.

Images