Method and device for executing in-memory calculation and storage system

Abstract

The invention relates to a method and device for executing in-memory computing and a storage system. A memory transistor array is used as a carrier of a weight matrix of a neural network in a flash memory that performs in-memory computation. The input data of the neural network is set to be transmitted to a series of bit lines of the memory transistor array, and the output data of the neural network is set to be captured from a series of source lines of the memory transistor array. Wherein the size of the weight of any transistor in the storage transistor array is adjusted by utilizing the threshold voltage of the transistor, so that the weight matrix of the neural network is dynamically configured.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of data storage, and more specifically, to an executable in-memory computing method and an executable in-memory computing device and a corresponding storage system in the field of data storage. Background Art

[0002] In the past, computing and storage were separated in electronic devices. The computing part, such as the processor, and the storage part, such as the hard disk or memory, were focused on their own tasks, and computing and storage basically did not interfere with each other. Some manufacturers also integrated the processor and memory closer in physical location in order to improve computing speed and computing power. For example, they were close to each other in the layout design of the circuit board, with short circuits, and directly stacked on each other during the packaging stage.

[0003] The industry's computing power has been gradually improved in accordance with the pace of Moore's Law, but the mainstream computing paradigm has always followed the von Neumann architecture. As a new type of computing power, the industry mainly focuses on the narrow sense of storage and computing integration, that is, using storage media for computing, but the computing media method has not yet been completely unified. As a new type of computing power, storage and computing integration is expected to solve the compilation wall, storage wall, and power consumption wall problems under the traditional von Neumann architecture. The consensus of the industry is that storage and computing integration has basically been identified as one of the key technologies to solve the computing power problem.

[0004] Integrated storage and computing organically integrates storage and computing. Based on its huge potential to improve energy efficiency, it is expected to become an advanced computing architecture solution in the computing power era. However, there are doubts: how to map the memory to the neural network and how to configure the neural network with a dynamically adjustable weight matrix to adapt to the actual application of the neural network such as back propagation, etc. This is a key issue that must be addressed when using in-memory computing. In particular: how to deal with the weight offset faced when updating the weights of the neural network inside the memory, which is a hidden problem for the memory. Summary of the invention

[0005] The present application relates to a method for performing in-memory computing, characterized by comprising:

[0006] A storage transistor array is used in a flash memory as a carrier of a weight matrix of a neural network. The input data of the neural network is set to be transmitted to a series of bit lines of the storage transistor array, and the output data of the neural network is set to be captured from a series of source lines of the storage transistor array. The weight of any transistor in the storage transistor array is adjusted using its threshold voltage, thereby dynamically configuring the weight matrix of the neural network.

[0007] In the above method, the storage transistor array used for the neural network is located on the same physical block of the flash memory; and the storage space where the input data and output data of the neural network are located and the storage space where the storage transistor array is located belong to different physical blocks of the flash memory.

[0008] In the above method, the original input data of the neural network is converted into input quantities in the form of voltage values, so as to be transmitted to a series of bit lines of the storage transistor array based on the input quantities sorted into vectors.

[0009] In the above method, the output data of the neural network is represented by the output current of the storage transistor array, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

[0010] The above method adjusts the weight of any transistor by changing its floating gate injection charge, and adjusts the weight of any transistor at its position in the storage transistor array (that is, at an element position corresponding to the any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

[0011] The above method dynamically configures the weight matrix of the neural network by resetting the weight size at each position in the storage transistor array during the back propagation stage of the neural network.

[0012] In the above method, a digital-to-analog converter is configured at each bit line of the storage transistor array, which is used to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line.

[0013] In the above method, an analog-to-digital converter is configured at each source line of the storage transistor array, which is used to convert the output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

[0014] The present application relates to a device for performing in-memory computing, characterized in that it comprises:

[0015] A flash memory, wherein the memory transistor array is used as a carrier of the weight matrix of the neural network, and the input data of the neural network is set to be transmitted to a series of bit lines of the memory transistor array, and the output data of the neural network is set to be captured from a series of source lines of the memory transistor array;

[0016] An input unit that converts raw input data of the neural network into an input quantity in the form of a voltage value to be transmitted to a series of bit lines of the storage transistor array based on the input quantity sorted into a vector;

[0017] An output unit is used to convert an output signal in the output current format at the source line into output data compatible with the digital output quantity of the neural network, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

[0018] The above-mentioned device for performing in-memory calculations, wherein the storage transistor array used for the neural network is located on the same physical block of the flash memory; and the storage space where the input data and output data of the neural network are located and the storage space where the storage transistor array is located belong to different physical blocks of the flash memory.

[0019] The above-mentioned device for performing in-memory calculations, wherein the original input data of the neural network is converted into input quantities in the form of voltage values, so as to be transmitted to a series of bit lines of the storage transistor array based on the input quantities sorted into vectors.

[0020] In the above-mentioned device for performing in-memory calculations, the output data of the neural network is represented by the output current of the storage transistor array, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

[0021] The above-mentioned device for performing in-memory calculations, wherein the method of adjusting the weight of any transistor includes changing its floating gate injection charge, and adjusting the weight of any transistor at the position where it is located in the storage transistor array (that is, at an element position corresponding to any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

[0022] In the above-mentioned device for performing in-memory calculations, the back propagation stage of the neural network can dynamically configure the weight matrix of the neural network by resetting the weight size at each position in the storage transistor array.

[0023] The above-mentioned device for performing in-memory calculations, the input unit includes at least a digital-to-analog converter, and a digital-to-analog converter is configured at each bit line of the storage transistor array, which is used to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line.

[0024] The above-mentioned device for performing in-memory calculations, the output unit includes at least an analog-to-digital converter, and an analog-to-digital converter is configured at each source line of the storage transistor array, which is used to convert the output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

[0025] The present application relates to a storage system for performing in-memory computing, characterized in that it includes:

[0026] A memory chip based on a flash memory, in which a memory transistor array is used as a carrier of a weight matrix of a neural network, wherein the weight of any transistor is adjusted by its threshold voltage, thereby dynamically configuring the weight matrix of the neural network; setting the input data of the neural network to be transmitted to a series of bit lines of the memory transistor array and also setting the output data of the neural network to be captured from a series of source lines of the memory transistor array;

[0027] A digital-to-analog converter built into the memory chip, wherein a digital-to-analog converter is configured at each bit line of the memory transistor array to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line;

[0028] An analog-to-digital converter is built into the memory chip, wherein an analog-to-digital converter is configured at each source line of the memory transistor array to convert an output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

[0029] The above-mentioned storage system, wherein the storage transistor array used for the neural network is located on the same physical block of the flash memory; and the storage space where the input data and output data of the neural network are located and the storage space where the storage transistor array is located belong to different physical blocks of the flash memory.

[0030] In the above-mentioned storage system, the original input data of the neural network is converted into an input quantity in the form of a voltage value, and is transmitted to a series of bit lines of the storage transistor array based on the input quantity sorted into a vector.

[0031] In the above storage system, the output data of the neural network is represented by the output current of the storage transistor array, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

[0032] In the above-mentioned storage system, the method of adjusting the weight of any transistor includes changing its floating gate injection charge, and adjusting the weight of any transistor at its position in the storage transistor array (that is, at an element position corresponding to the any transistor in the weight matrix) by changing the floating gate injection charge of the transistor.

[0033] The above-mentioned storage system dynamically configures the weight matrix of the neural network by resetting the weight size at each position in the storage transistor array during the back propagation stage of the neural network.

[0034] In the above-mentioned storage system, each bit line of the storage transistor array is configured with a digital-to-analog converter, which is used to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line.

[0035] In the above-mentioned storage system, each source line of the storage transistor array is equipped with an analog-to-digital converter, which is used to convert the output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

[0036] The present application relates to a chip for performing in-memory computing, characterized in that it includes:

[0037] A storage transistor array, which is used as a carrier of a weight matrix of a neural network. The weight of any transistor is adjusted by using its threshold voltage, thereby dynamically configuring the weight matrix of the neural network; setting the input data of the neural network to be transmitted to a series of bit lines of the storage transistor array, and setting the output data of the neural network to be captured from a series of source lines of the storage transistor array;

[0038] A digital-to-analog converter integrated in the chip, wherein a digital-to-analog converter is configured at each bit line of the storage transistor array to convert input data of the neural network into a voltage value that matches the input signal pattern of the bit line;

[0039] An analog-to-digital converter is integrated in the chip, wherein an analog-to-digital converter is configured at each source line of the storage transistor array to convert the output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

[0040] Based on the foregoing, one of the greatest advantages of the present application is that the weight matrix of the neural network is directly mapped to the storage transistor array and the transistor threshold voltage can be adjusted to dynamically configure the weight matrix of the neural network so as to adapt to the weight matrix update of the neural network, such as matching back propagation. This can also cope with the more difficult problems such as weight offset faced when updating the weights of the neural network inside the memory, and prevent one or more elements in the weight matrix mapped by the memory from having unexpected hidden offsets.

[0041] Based on the above, the second biggest advantage of this application is that it adopts a non-von Neumann architecture, which can solve the problems of compilation wall, storage wall, and power consumption wall under the traditional von Neumann architecture. One convenient aspect of compilation is to realize the programming of core weights by adjusting the threshold voltage of transistors. This solution only needs to configure the voltage or current of the three terminals of the transistor at the transistor to realize the compilation of core weights. Because it is a compilation compatible with information such as voltage and current on a traditional memory, there is no obstacle in designing a part of the compilation behavior. Moreover, unlike the traditional solution, which requires data exchange between the memory and the processor and the subsequent storage wall, the data storage and data calculation of this application, especially matrix operations, occur inside the memory, solving the problem of storage wall. Since the specific implementation of the neural network is based on the premise that there is no need to implement a large amount of data transfer between the memory and the processor, and since most of the energy consumption of the power consumption wall is generated during the data transfer process, for example, the data transfer power consumption is more than a thousand times the computing power consumption, it also solves the problem of high energy consumption in the application scenarios of neural networks and their artificial intelligence. In addition, AI computing requires 1PB / s, while data transfer speeds such as DRAM 40GB-1TB / s are far from sufficient to meet the needs of AI computing. If internal memory calculations as in this application are used, the concern that the data transfer speed cannot keep up with the AI ​​computing speed can be better resolved.

[0042] Based on the foregoing, the third biggest advantage of this application is to promote a significant increase in computing power. By using storage-computing integrated technology, the weight part of a large number of multiplication and addition calculations in AI calculations can be stored in the storage unit, and modifications can be made to the existing storage circuit part of the storage unit, so that data input and calculation processing can be performed while reading, and neural networks such as convolution operations are completed in the storage array. Since convolution operations with a large number of multiplications and additions are the core components of deep learning, in-memory computing and in-memory logic are very suitable for deep neural network applications of artificial intelligence and big data technologies based on AI. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the above purposes, features and advantages more understandable, the following text will explain the specific implementation methods in detail in conjunction with the drawings. After reading the following description and referring to the following drawings, the features and advantages of the present application will become apparent.

[0044] Figure 1 A part of the transistors inside the flash memory are selected as the weight carriers of the neural network.

[0045] Figure 2 The input data to the neural network comes from a series of bit lines in the storage transistor array.

[0046] Figure 3Taking a picture or video as an example, pixel values ​​or grayscale values ​​are input into a storage transistor array.

[0047] Figure 4 It maps the weight matrix of the neural network to the storage transistor array of the flash memory.

[0048] Figure 5 The weight value of the transistor in the transistor array is adjusted by changing the charge injected into the floating gate.

[0049] Figure 6 It is to deal with concerns such as weight offset when updating the weights of the neural network inside the memory. DETAILED DESCRIPTION

[0050] The technical solutions disclosed in the present application will be clearly and completely described below in conjunction with specific embodiments. However, the embodiments described are only embodiments used for narrative explanation in the present application and are not all embodiments. Based on these embodiments, technicians in this field should realize that any solution obtained without creative work falls within the scope of protection of the present application.

[0051] See also Figure 1 This is a circuit of an existing general-purpose NOR flash memory. Its working principle is that a floating gate (Floating Gate) for storing charges such as electrons can be formed between the source S and the drain D of the storage transistor, that is, on a semiconductor where current is conducted in one direction. The floating gate is wrapped with a layer of silicon oxide film insulator. Above the floating gate is a selection / control gate (Control Gate) that controls the conduction current between the source and the drain. The charges on the floating gate, such as electrons, will not disappear due to power failure, so flash memory is a non-volatile memory. The write operation, such as applying a positive voltage to the control electrode, allows charges such as electrons to pass through the insulating layer into the floating gate. The erase operation is just the opposite, such as applying a positive voltage to the substrate to separate electrons from the floating gate to achieve erasure. The data operation of flash memory belongs to the prior art, and this article will not repeat it separately. The read, write or erase operations of flash memory in the prior art can be applied to the flash memory of this application.

[0052] See also Figure 1 , describes flash memory in array form: the gate of each row of transistors is connected to a common word line and the drain of each column of transistors is connected to a common bit line and the source of each column of transistors is connected to a common source line.

[0053] See also Figure 1 , with respect to the row word line example, the gates of the first row transistors are connected to the word line G1 and the gates of the second row transistors are connected to the word line G2, and the gates of the Mth row transistors are connected to the word line GM. More omitted rows are not repeatedly shown in the figure. The total of M rows of transistors is regarded as the row element of the matrix M×N.

[0054] See also Figure 1 Regarding the row source line example, the sources of the first row transistors are connected to the source line S1, the sources of the second row transistors are connected to the source line S2, and the sources of the Mth row transistors are connected to the source line SM. More omitted rows are not repeatedly shown in the figure.

[0055] See also Figure 1 For the example of column bit lines, the drains of the first column transistors are connected to the bit line B1, the drains of the second column transistors are connected to the bit line B2, and the drains of the Nth column transistors are connected to the bit line BN. More omitted columns are not shown repeatedly in the figure. The total N columns of transistors are regarded as column elements of the matrix M×N.

[0056] See also Figure 1 The original function of the flash memory shown in the figure is to store code elements 1 or 0. In other words, the flash memory has no special use in traditional computers, servers or other similar electronic devices. However, the present application no longer regards the flash memory as a storage space that only stores code elements 1 or 0, but advocates changing the circuit architecture of the flash memory to a computing unit that can participate in artificial intelligence, and the underlying physical structure of the flash memory remains almost unchanged.

[0057] See also Figure 2 , using flash memory-based computing units to replace traditional computers is because there are several major drawbacks in the operation phase of computers executing neural networks: reading and writing data from the memory outside the processor, the data transfer time is hundreds or thousands of times the operation time, the power consumption of the whole process is about 60%-90%, the energy efficiency is very low, and the incidental storage wall problem has become a major obstacle to data computing applications. In particular, the biggest challenge for accelerating deep learning in neural networks is the frequent movement of data between computing units and storage units. Although multi-core CPUs or multi-core GPUs are used to improve computing power, in the post-Moore era, storage bandwidth restricts the effective bandwidth of computing systems and makes the overall computing power growth difficult. Running weight calculations of neural networks or running inference calculations of neural networks on traditional computers is time-consuming and power-consuming, which is one of the most prominent manifestations of this drawback.

[0058] See also Figure 2, the neural network has encountered a bottleneck in its development due to the introduction of a large number of parameters. Each output neuron of the fully connected layer is connected to all input neurons. According to the assumptions in the figure: if the number of input neurons is N and the number of output neurons is M, then there are at least approximately N×M weights or M biases that need to be trained and calculated. For each output neuron of the convolutional layer, there are usually input neurons within the coverage of the convolution kernel connected to it, and the convolution kernel used for all outputs is the same. Although local connections and weight sharing reduce the parameters of the convolutional layer, a large amount of calculations still need to be performed under the conditions of large-size image input and deep neural networks. The reading and calculation of large amounts of data face the memory bottleneck and computing bottleneck of the network. The power consumption and latency of neural networks have risen sharply, limiting the application of neural networks.

[0059] See also Figure 2 In order to solve the aforementioned bottleneck, the non-von Neumann architecture shown in the figure can provide greater computing power and higher energy efficiency in specific fields, and can surpass existing ASIC computing chips. In-memory computing and in-memory logic, that is, the integrated storage and computing technology directly uses the memory for data processing or calculation, thereby integrating data storage and data calculation in the same chip or the same area of ​​the same memory, which can completely eliminate the bottleneck of the von Neumann computing architecture, especially for large-scale parallel application scenarios such as deep learning neural networks with large amounts of data.

[0060] See also Figure 2 In the design, bit lines B1-BN are used as input and source lines S1-SM are used as output. Figure 1 The flash memory is regarded as the physical carrier that can realize the neural network NET.

[0061] See also Figure 2 In order to explain the specific implementation process, we first design a matrix Ma_wei as follows.

[0062]

[0063] See also Figure 2 , the input of bit lines B1-BN is represented by the vector Ma_in=[B1, B2...BN]. For the convenience of mathematical expression, the input of bit line B1 needs to be mapped to X1, the input of bit line B2 needs to be mapped to X2, the input of bit line B3 needs to be mapped to X3, and the input of bit line BN needs to be mapped to X N .

[0064] See also Figure 2 , the output of source line S1-SM is represented by vector Ma_out = [S1, S2...SM]. For the convenience of mathematical expression, the output of source line S1 needs to be mapped to Y1, the output of source line S2 needs to be mapped to Y2, the output of source line S3 needs to be mapped to Y3, ..., the output of source line SM needs to be mapped to YM .

[0065] See also Figure 2 The purpose of the flash memory is to perform the computing function of the neural network NET. The specific computing process is expressed by the following formula. In fact, this process is also a detailed example of simulating the flash memory as a computing unit or a quasi-computing unit. M etc. as representatives to illustrate this spirit.

[0066] Y1=X1*W 11 +X2*W 12 +X3*W 13 +…X N *W 1N

[0067] Y2=X1*W 21 +X2*W 22 +X3*W 23 +…X N *W 2N

[0068] Y M =X1*W M1 +X2*W M2 +X3*W M3 +…X N *W MN

[0069] See also Figure 2 A more intuitive expression is to use vector-matrix multiplication to show the aforementioned complex operation. The vector-matrix expression flash memory performs local data processing of the neural network NET as follows.

[0070]

[0071] See also Figure 2 If we define X1 to X N is the input layer of the neural network NET, then Y1 to Y N The corresponding layer is the output layer of the neural network NET. Figure 1 The flash memory completes the entire vector operation from the input layer to the output layer, and this operation is very different from the operation of the traditional von Neumann computer, because its operation only needs to measure the respective signal quantities from the source lines S1-SM, and the single multiplication and addition do not need to be implemented one by one. The final operation result can be fully reflected from the signal quantities of the source lines S1-SM at one time. So it is called quasi-computing. The operation of the traditional von Neumann computer must implement all multiplications and additions one by one, which is the biggest difference between the two.

[0072] See also Figure 2, the neural network NET based on flash memory is suitable for the field of image processing. Assume that data such as pictures or videos with pixel values ​​or grayscale values ​​(artificial intelligence, autonomous driving, face recognition, target detection) are input into the neural network NET based on flash memory, and the image Gr_in is taken as an example below.

[0073] See also Figure 3 , the pixels in the image Gr_in are divided into multiple rows and columns, and the pixel corresponding to the address of the 0th pixel is defined as g1, the pixel corresponding to the address of the 1st pixel is defined as g2, the pixel corresponding to the address of the 2nd pixel is defined as g3, and so on. The pixel corresponding to the last pixel address is gn. According to the arrangement rule and address regulations of the pixel matrix, the gray value of any pixel can be obtained by the image obtained by shooting. Allow the gray value or pixel value of n pixels (belonging to the image Gr_in) to be coupled to the X1 to X N Of course, the grayscale value or pixel value can also be normalized or similar image data processing can be performed.

[0074] See also Figure 3 , one issue worth noting is that the input quantity format of the bit lines B1-BN should conform to the inherent characteristics of the flash memory, that is, the input value of the bit lines B1-BN is not directly equivalent to the pixel value g1-gn. Therefore, in an optional embodiment, the pixel values ​​g1-gn are converted from digital to analog by a DAC device such as a digital-to-analog converter packaged together with the flash memory, and the analog quantities corresponding to the pixel values ​​g1-gn are then input to the bit lines B1-BN. The DAC (Digital to Analog Converter) device can also be directly integrated with the flash memory on the same semiconductor chip during the wafer manufacturing stage. At this point, the data contained in the image Gr_in can be smoothly coupled with the flash memory. Another important factor is that the final storage location or space of the image Gr_in must usually be various types of memory. Then the storage location of the image Gr_in and the location where the matrix operation is performed are the same flash memory. The outstanding advantage of this inherent characteristic can be revealed. The processor CPU or GPU will no longer need to read data from the memory to itself to perform matrix operations. The storage and calculation actions are all implemented in the memory. The concerns about reading and writing data from the processor to the memory, the time cost of data transportation, etc. mentioned in the previous article will no longer exist.

[0075] See also Figure 3 There is an important problem that has not been solved: In order to explain the specific implementation process of the neural network, the previous article specially designed the matrix Ma_wei involved in the network calculation. However, how to map the M*N vector matrix Ma_wei to the transistors in the flash memory? Figure 1 The M×N elements shown still present concerns that need to be resolved.

[0076] See also Figure 4 , regarding the relationship between pixel values ​​g1-gn and bit lines B1-BN, pixel value g1 is converted into voltage value V1 through DAC digital-to-analog conversion, pixel value g2 is converted into voltage value V2 through digital-to-analog conversion, pixel value g3 is converted into voltage value V3 through digital-to-analog conversion, and so on, pixel value gn is converted into voltage value VN through digital-to-analog conversion. According to the interface correspondence shown in the figure, voltage value V1 is connected to bit line B1, voltage value V2 is connected to bit line B2, voltage value V3 is connected to bit line B3, and so on, voltage value VN is connected to bit line BN. According to the rule, the larger the pixel value, the larger the converted bit line voltage value, and the smaller the pixel value, the lower the converted bit line voltage value. Furthermore, the number of transistor layouts, that is, the size of the matrix M×N, should have a corresponding relationship with the pixel size of the image Gr_in.

[0077] See also Figure 4 , with respect to the matrix Ma_wei, the gates of the transistors in the first row are connected to the word line G1, the gates of the transistors in the second row are connected to the word line G2, and the gates of the transistors in the Mth row are connected to the word line GM. The weights generated by the transistors in the first row of the weight matrix are recorded as W 11 , W 12 , W 13 , ...W 1N The second row of transistors in the weight matrix have their own weights, which are recorded as W. 21 , W 22 , W 23 , ...W 2N The weights generated by the Mth row of transistors in the weight matrix are recorded as W. M1 , W M2 , W M3 , ...W MN . So far, each transistor is configured with a corresponding weight value that contributes to the matrix Ma_wei involved in the network calculation. Traditional transistors themselves do not have features associated with weights, especially flash memory and neural networks have no direct relevance, so how to map weights to transistors is another key problem that needs to be solved.

[0078] See also Figure 4, the total of M rows and N columns of transistors are regarded as M×N matrix elements. The definition of each weight value in the matrix Ma_wei is the basis for realizing the neural network NET. The figure shows that the source current of each row of transistors flows to the common source line of the row of transistors, and the figure shows that the drain of each column of transistors is connected to the common bit line of the column of transistors. This layout of transistors will provide a feasible solution for how to define the weights of transistors. For example, the source current convergence characteristics of each row of transistors correspond to the multiplication and accumulation calculation algorithm of the vector matrix multiplication mentioned above.

[0079] See also Figure 4 To explain how to map weights to transistors, let’s first introduce the relationship between the current and threshold of the storage transistor and the applied voltage: The threshold voltage of the gate is V TH ; The process parameter K of the storage transistor related to the process is μCW / L, where W / L is the aspect ratio of the transistor. For a storage transistor with a certain manufacturing process and a certain size, the process parameter K is almost a constant; the voltage between the gate and the source is V GS and the voltage V between the transistor drain and source DS Can satisfy K(V GS -V TH )V DS = I. When the current I flowing through the transistor operates in the linear region or the triode region, if the current I of a single transistor performs multiplication and the output currents I of a row of transistors perform accumulation, this is equivalent to the multiplication and accumulation calculation of the matrix multiplication.

[0080] See also Figure 4 , for the first row of transistors (gate coupled to G1): the current I1 of the first transistor is approximately K(V GS1 -V TH1 )V DS1 =I1, according to the same current formula, the current I2 of the second transistor is calculated to be approximately K(V GS2 -V TH2 )V DS2 =I2, Nth transistor current K(V GSN -V THN )V DSN =IN. The voltage between the gate and source of the first transistor is V GS1 The voltage between the drain and source is V DS1 , the threshold voltage is V TH1 , the voltage between the gate and source of the second transistor is V GS2 The voltage between the drain and source is V DS2 , the threshold voltage is V TH2 , the voltage between the gate and source of the Nth transistor is V GSNThe voltage between the drain and source is V DSN , the threshold voltage is V THN .

[0081] See also Figure 4 For the first row of transistors (gate coupled to G1): the source currents of the first transistor to the Nth transistor all flow to the common source line S1 of the transistors in the row, and the current of the source line S1 is recorded as IS1.

[0082] IS1=I1+I2+I3+…IN

[0083] See also Figure 4 , for the first row of transistors (gate coupled to G1): if W is defined at the location of the first transistor 11 With K(V GS1 -V TH1 ) are proportional or equal, and V1 and V DS1 If the current I1 flowing through the first transistor is proportional or equal, the current I1 can be expressed as I1 = W 11 × V1. If W is defined at the location of the second transistor 12 With K(V GS2 -V TH2 ) are proportional or equal, and V2 and V DS2 If the current I2 flowing through the second transistor is proportional or equal, the current I2 can be expressed as I2 = W 12 × V2. If W is defined at the location of the third transistor 13 With K(V GS3 -V TH3 ) are proportional or equal, and V3 and V DS3 If the current I3 flowing through the third transistor is proportional or equal, the current I3 can be expressed as I3 = W 13 ×V3. If W is defined at the location of the Nth transistor 1N With K(V GSN -V THN ) are proportional or equal, and VN and V DSN If the current IN flowing through the Nth transistor is proportional or equal, the current IN can be expressed as IN = W 1N ×VN.

[0084] See also Figure 4, for the first row of transistors (gate coupled to G1): the current of source line S1 is recorded as IS1, which can be expressed as follows after the conversion of the formula (Y1=IS1 can also be defined). The prerequisite is that the source current of the first row of transistors has the characteristic of converging, which corresponds to the multiplication and accumulation calculation algorithm of vector-matrix multiplication. Therefore, it is one of the main factors that flash memory can replace some functions of traditional CPU or GPU.

[0085] IS1=Y1=W 11 *V1+W 12 *V2+W 13 *V3+…W 1N *VN

[0086] See also Figure 4 , for the second row of transistors (gate coupled to G2): the current of the source line S2 is denoted as IS2 and can be expressed in the following way after conversion of the formula (Y2=IS2 can also be defined).

[0087] IS2=Y2=W 21 *V1+W 22 *V2+W 23 *V3+…W 2N *VN

[0088] See also Figure 4 , for the Mth row of transistors (gate coupled to GM): the current of the source line SM is recorded as ISM, which can be expressed as follows after the conversion of the formula (Y can also be defined M =ISM).

[0089] ISM=Y M =W M1 *V1+W M2 *V2+W M3 *V3+…W MN *VN

[0090] See also Figure 4 , as defined above, X1 to X N As the input layer of the neural network NET, define Y1 to Y N The corresponding is the output layer of the neural network NET. So far, the first row of transistors in the flash memory has completed a vector operation Y1 from the input layer to the output layer, and the result of Y1 can be directly calculated at one time by sensing the current of the source line S1 of the first row of transistors, without having to calculate W separately. 11 ×V1 or W 12 ×V2 or W 1N×VN, etc., belong to the class calculation under the non-von Neumann architecture. The calculation result is not obtained through numerical calculation, but through current sensing. Similarly, the second row of transistors in the flash memory completes a vector operation Y2 from the input layer to the output layer, the third row of transistors in the flash memory completes a vector operation Y3 from the input layer to the output layer, and the Mth row of transistors in the flash memory completes a vector operation Y from the input layer to the output layer. M In an optional embodiment, the current IS1-ISM is converted from analog to digital by an ADC device such as an analog-to-digital converter packaged together with the flash memory, and the digital quantities corresponding to the current values ​​are suitable for various electronic devices. The ADC (Analog to Digital Converter) device can also be directly integrated with the flash memory on the same semiconductor chip during the wafer manufacturing stage.

[0091] See also Figure 4 One of the tasks performed by the transistor array of the flash memory is to construct the matrix Ma_wei, and each element in the matrix corresponds to the weight matrix of the neural network NET.

[0092] See also Figure 4 , the matrix Ma_wei takes the neural network NET input layer Lay1 / hidden layer Lay2 as an example. When the input data is input to the input layer Lay1, there is a matrix Ma_wei between the input layer Lay1 and the hidden layer Lay2, and the input data is converted into voltage V1-VN to give to the input layer Lay1, then the output obtained by the hidden layer Lay2 is equivalent to the input data obtained by performing matrix operations by the weight matrix Ma_wei. The voltage signal is the type of input and the output data is the current value signal. The weight matrix Ma_wei affects the input voltage signal and thus affects the amplitude of the output current value. This is also the general process of realizing artificial intelligence operations through NOR Flash memory.

[0093] See also Figure 4 In addition to being used for AI computing, flash memory can also be used for other computing applications such as brain-like chips. It represents a future mainstream big data computing chip architecture, and can also be used as a core computing module for low-energy supercomputing centers. For example, NOR FLASH in-memory computing can support about 300M of deep learning weight parameters in a single chip, and can perform calculations without additional memory.

[0094] See also Figure 4 Compared with other types of general-purpose computing, flash memory has several important characteristics described below for neural network NET computing, which are also its own advantages.

[0095] See also Figure 4 First, the calculation process of the neural network NET based on flash memory is simple and allows a large amount of parallelism. The calculation of the neural network NET no longer uses the branch prediction and speculative execution mechanisms of the CPU used in general computing to run efficiently. This feature allows the flash memory hardware accelerator for the neural network NET to reduce or break away from the use of CPU or GPU as much as possible, thereby performing low-cost and high-efficiency calculations.

[0096] See also Figure 4 , Secondly, the calculations performed by the neural network NET consist of only a few single operations, such as matrix multiplication and vector operations, the application of convolution kernels, and other linear algebra operations. This makes the neural network NET accelerator similar to the GPU, using a large number of simple computing units to cope with the complex data operations that the industry needs to perform.

[0097] See also Figure 4 ,Thirdly, the neural network NET itself shows a great sparsity feature, and a large number of its parameters and calculation results are very small, or even close to zero. Utilizing the sparsity feature can make the operation reasonably use the data to skip the calculation of multiplying by zero, thereby improving the overall calculation speed, especially matrix operations.

[0098] See also Figure 4 Fourthly, the neural network NET allows for low-precision calculations such as low-bit precision. For example, for most artificial intelligence applications, 8-bit or 4-bit (meaning a small transistor array) fixed-point numbers can be used for quantization without losing recognition accuracy. Low-precision fixed-point numbers can be used for calculations based on artificial intelligence application scenarios to increase reasoning speed and reduce power consumption. Power loss is a point worth emphasizing. Traditional CPUs / GPUs generate a lot of heat during model training and reasoning stages, and most of the power is lost in the heat dissipation of the processor. The class calculation of this application completely avoids the processor's accumulation process for monocular multiplication and multiplication results, and reduces unnecessary data transfer between the processor and the memory, reducing energy consumption to 1 / 10 to 1 / 100 of the original.

[0099] See also Figure 4 For example, the size of intelligent voice models is usually hundreds of KB to several megabytes, and the size of image inference models on the end side is usually between several megabytes and tens of megabytes. The storage space of conventional NOR FLASH on the market can easily handle models of this size. Therefore, even the NOR FLASH in-memory computing chip of a handheld terminal is sufficient to meet the needs of most AI scenarios. Under the premise of optimizing the NOR FLASH structure, such as the series-parallel or cascade connection of many flash memories, it is almost comparable to all artificial intelligence application scenarios that can be achieved by GPUs.

[0100] See also Figure 5 , the previous article focuses on weights such as W 11 and transistor parameters such as K(V GS1 -V TH1 ) is explained. In the neural network NET, weights are information that needs to be adjusted and updated at any time, so the part that deserves detailed explanation also includes how to adaptively modify transistor parameters (or weights). In particular, neural networks include back propagation, and the core of back propagation is the update and iteration of weights. Obviously, the local calculation of flash memory is different from the weight iteration of traditional processors.

[0101] See also Figure 5 , the threshold voltage of the transistor is V TH , if the threshold voltage is changed, then it is equivalent to K(V GS -V TH ) is changed, which is equivalent to changing K(V GS -V TH ) is proportional to the weight value, so changing the threshold voltage V TH It is an optional method. The writing and erasing of flash memory data is performed by manipulating the charge inside the floating gate related to the control gate, such as charge release and tunneling. For example, in the writing method, NOR Flash uses hot electron injection. By increasing the voltage of the control gate during writing, such as injecting charge from the source to the floating gate, the change in the charge amount of the floating gate can change K (V GS -V TH ). For example, in the erasing method, NOR Flash uses the tunnel effect to remove the charge in the floating gate, for example, the voltage applied to the control gate is a negative voltage.

[0102] See also Figure 5 Taking the transistors in the Mth row (the gate is coupled to the word line GM) as an example, in the erase operation of programming PG0, the transistors determined by the bit line BN (with a weight of W) can be cleared. MN ) floating gate charge, so its threshold voltage is reduced and the weight W MN The erase of programming PG0 is typically performed by applying a negative voltage to the word line GM to cause the floating gate charge of the transistor to generate a quantum tunnel effect and enter the substrate where the transistor is located. According to the requirements of the weight value modification mentioned above, in contrast, in the write action of programming PG1, for example, a transistor determined by the bit line BN (with a weight of W) can be injected. MN ) floating gate charge, in this case, so the threshold voltage of the transistor is increased and the weight W MN Furthermore, in the write operation of programming PG2, the transistor (weight W) determined by the bit line BN is continuously injected.MN ) floating gate charge, programming PG2 is a fine-tuning compared to programming PG1, so the threshold voltage of the transistor continues to increase and the weight W MN The voltage of charge injection programming PG1 is much higher than that of fine-tuning programming PG1. The small voltage scale of programming PG2 is to avoid over-adjustment of weight. In fact, charge injection programming can adopt more injection times than those in the figure, so that the weight W MN Not only can the multi-value requirement of the weight matrix Ma_wei be met, but also the high-precision requirement of the weight value can be met. The execution of multiple cycles of the erase programming action and the charge injection programming action as shown in the figure represents the iteration of the weight.

[0103] See also Figure 4 In an optional embodiment, the method for performing in-memory calculations includes: using a storage transistor array as a carrier of a weight matrix of a neural network NET in a flash memory, setting the input data of the neural network to be transmitted to a series of bit lines B1-BN of the storage transistor array, and also setting the output data of the neural network NET to be captured from a series of source lines S1-SM of the storage transistor array. The weight of any transistor in the storage transistor array is adjusted by its threshold voltage, thereby dynamically configuring the weight matrix of the neural network. For example, a transistor T determined by the bit line B1 and the source line S1 11 :Transistor T 11 The weight of the transistor T is adjusted by its threshold voltage. 21 :Transistor T 21 The weight of the transistor T is adjusted by its threshold voltage. 31 :Transistor T 31 The weight of the transistor T is adjusted by its threshold voltage. The bit line B1 and the source line SM determine M1 :Transistor T M1 The weight of any transistor is adjusted by its threshold voltage. Therefore, the weight of any transistor is adjusted by its threshold voltage, thereby dynamically configuring the weight matrix of the neural network. N and M are positive integers. For example, the transistor T determined by the bit line BN and the source line SM MN :Transistor T MN The weight is adjusted by its threshold voltage, the charge on the floating gate, etc.

[0104] See also Figure 4In an optional embodiment, the storage transistor arrays (transistor arrays) for the neural network determined by the bit lines B1-BN and the source lines S1-SM are located on the same physical block (block) of the flash memory; and the space where the input data (such as vector Ma_in) and the output data (such as vector Ma_out) of the neural network are located respectively and the storage space where the storage transistor arrays (transistor arrays) are located belong to different physical blocks (blocks). For example, the storage space where the input data is located and the storage space where the output data is located belong to the same physical block, or the storage space where the input data is located and the storage space where the output data is located belong to the same physical block respectively. However, the storage space where the input data and the output data are located respectively and the storage space where the storage transistor array is located belong to different physical blocks. The stripping of this storage space is mainly to allow the storage class space and the computing class space to have different erase operations and write operations, which is equivalent to using the threshold voltage for weight adjustment ( Figure 5 The weight configuration operation of the PG0, PG1, PG2, etc.) and the data read and write operation of the storage data, such as the input and output data of the neural network, are performed in different storage spaces, and the weight configuration operation and the storage data read and write operation do not interfere with each other. This is because the programming voltage (or programming current) used in the weight configuration operation and the storage data read and write operation requires different voltage levels (or current levels).

[0105] See also Figure 4 In an optional embodiment, the original input data of the neural network NET is converted into an input quantity in the form of a voltage value, and is transmitted to a series of bit lines of the storage transistor array based on the input quantity sorted into a vector. The original input data includes data types such as pixel values ​​or grayscale values ​​of an image or video, and the original input data is converted into input quantities V1, V2, V3...VN in the form of voltage values ​​as shown in the figure. The voltage value V1 is input to the bit line B1, the voltage value V2 is similarly input to the bit line B2, the voltage value V3 is input to the bit line B3, and the voltage value VN is input to the bit line BN.

[0106] See also Figure 4 The input is represented by a vector Ma_in=[B1=V1, B2=V2...BN=VN]. The input quantity [V1, V2...VN] is sent to a series of bit lines B1-BN of the array (transistor arrays) based on the input quantity [V1, V2...VN] sorted into a vector.

[0107] See also Figure 4In an optional embodiment, the output data of the neural network NET is represented by the output current of the storage transistor array, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector. For example, the output currents of the series of source lines S1-SM of the storage transistor array are respectively recorded as IS1, IS2, ... ISM. The output current IS1 of the source line S1 can be converted into a digital quantity through an ADC analog-to-digital conversion, and the output current ISM of the similar source line SM can be converted into a digital quantity through an ADC analog-to-digital conversion. The output data of the network NET is represented by the output currents IS1, IS2, ... ISM of the storage transistor array.

[0108] See also Figure 4 In an optional embodiment, the output current of the series source lines S1-SM of the storage transistor array is regarded as the output quantity [IS1, IS2, ... ISM] of the storage transistor array arranged into a vector Ma_out. The output of the neural network can also be considered to be represented by the vector Ma_out = [S1 = IS1, S2 = IS2, ... SM = ISM].

[0109] See also Figure 5 In an alternative embodiment, the regulating transistor (T MN ) weights include changing the floating gate injection charge (such as changing T MN The floating gate charge of the transistor is changed to adjust the weight of any transistor at the position (such as the position determined by BN / SM) in the memory transistor array. For example, the weight of T MN The weight W MN The size.

[0110] See also Figure 5 In an optional embodiment, during the back propagation phase of the neural network NET, the weight matrix of the neural network is dynamically configured by resetting the weights at various locations in the storage transistor array. The various locations in the array mentioned here include each location where the bit lines B1-BN and the source lines S1-SM intersect each other. In fact, the various locations mentioned also represent the row and column indexes of each element in the weight matrix. For example, in the weight matrix W MN The row and column index of this element is determined by or corresponds to the position of the bit line BN / source line SM. In the back propagation algorithm, the neural network needs to repeatedly iterate the size value of each element in the weight matrix until it meets the requirements.

[0111] See also Figure 4In an optional embodiment, the input data of the neural network is transmitted to a series of bit lines of the storage transistor array and the output data of the neural network is captured from a series of source lines of the storage transistor array: the source of each row of storage transistors in the transistor array is connected and regarded as a source line output end of one output data of the neural network, and the drain of each column of storage transistors in the transistor array is connected and regarded as a bit line input end of one input data of the neural network. In addition, the gate of each row of storage transistors in the storage transistor array is connected and coupled to a word line.

[0112] See also Figure 3 In an optional embodiment, a digital-to-analog converter DAC is configured at each of the bit lines B1-BN of the storage transistor array, and the digital-to-analog converter DAC is used to convert the input data of the neural network NET into a voltage value that matches the input signal pattern of the bit line (for example, the bit line requires a voltage signal as an input quantity).

[0113] See also Figure 3 In an optional embodiment, an analog-to-digital converter ADC is configured at each of the source lines S1-SM of the storage transistor array. The analog-to-digital converter ADC is used to convert the output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network NET. For example, the analog-to-digital converter configured at the source line SM is used to convert the output signal in the output current ISM format at the source line SM into output data that is compatible with the digital output quantity of the neural network. The digital quantity (digital) of current is different from the analog quantity (analog). The analog-to-digital converter configured at the source line S1 is used to convert the output signal in the output current IS1 format at the source line S1 into output data that is compatible with the digital output quantity of the neural network. The same is true for the others S2 to SM-1.

[0114] See also Figure 4 In an optional embodiment, the device for performing in-memory calculations includes: a flash memory, an input unit, and an output unit. Regarding the flash memory portion of the device, its storage transistor array can be used as a carrier of the weight matrix of the neural network, and the input data of the neural network can be set to be transmitted to a series of bit lines of the storage transistor array, and the output data of the neural network can be set to be captured from a series of source lines of the storage transistor array. And regarding the input unit portion of the device, it can convert the original input data of the neural network into an input quantity in the form of a voltage value, so as to be transmitted to a series of bit lines of the storage transistor array based on the input quantity sorted into a vector. Regarding the output unit of the device, it is used to convert the output signal in the output current format of the source line into output data adapted to the digital output quantity of the neural network, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector. The input unit and the output unit respectively include a digital-to-analog converter and an analog-to-digital converter.

[0115] See also Figure 4 In an optional embodiment, the storage system for performing in-memory calculations includes: a storage chip based on flash memory and a digital-to-analog converter built into the storage chip and an analog-to-digital converter built into the storage chip. A storage chip based on flash memory, wherein a storage transistor array is used in the flash memory as a carrier of the weight matrix of the neural network and the weight of any transistor therein is adjusted using its threshold voltage, thereby dynamically configuring the weight matrix of the neural network; setting the input data of the neural network to be transmitted to a series of bit lines of the storage transistor array and also setting the output data of the neural network to be captured from a series of source lines of the storage transistor array. Regarding the digital-to-analog converter built into the storage chip, wherein a digital-to-analog converter is configured at each bit line of the storage transistor array, so that the input data of the neural network can be converted into a voltage value that matches the input signal pattern of the bit line. Regarding the analog-to-digital converter built into the storage chip, wherein an analog-to-digital converter is configured at each source line of the storage transistor array to convert the output signal in the output current format at the source line into output data that matches the digital output quantity of the neural network.

[0116] See also Figure 5 In the past, computing and storage were separated in electronic devices. Computing, such as the processor, and storage, such as hard disks or memory, were focused on their own tasks, and computing and storage basically did not interfere with each other. Some manufacturers also integrate processors and memories closer in physical locations in order to improve computing speed and computing power. For example, they are close to each other in the layout design of the circuit board, with short circuits, and are directly stacked on each other during the packaging stage.

[0117] See also Figure 5 , computing power has been gradually improved in accordance with the pace of Moore's Law, but the mainstream computing paradigm has always followed the von Neumann architecture. As a new type of computing power, the industry mainly focuses on the narrow sense of storage and computing integration, that is, using storage media for computing, but the computing media method has not yet been completely unified. As a new type of computing power, storage and computing integration is expected to solve the compilation wall, storage wall, and power consumption wall problems under the traditional von Neumann architecture. The consensus of the industry is that storage and computing integration has basically been identified as one of the key technologies to solve the computing power problem.

[0118] See also Figure 5In this application, the input data or output data of the neural network mapped to the storage transistor array can be the data of a certain layer (input layer / hidden layer / output layer, etc.) of the neural network. In other words, the storage transistor array is allowed to be only a local sub-network structure of the neural network. Alternatively, the neural network can use multiple storage transistor arrays as shown in the figure to perform a series-parallel cascade combination of the front and rear stages, and multiple storage transistor arrays are combined into a local sub-network structure of the neural network or into the entire network structure.

[0119] See also Figure 5 , storage and computing are integrated to organically integrate storage and computing. Based on its huge potential for improving energy efficiency, it is expected to become an advanced computing architecture solution in the computing power era. However, there are doubts: how to deal with the weight offset faced by the neural network weight update inside the memory (for example, W MN This is a hidden problem that is difficult to detect for memory, and will be explained in detail below.

[0120] See also Figure 6 It is known that the storage transistor arrays (transistor arrays) used for the neural network NET can be determined by the bit lines B1-BN and the source lines S1-SM. This array is used as an example to explain the problem of weight offset. The size of the storage transistor array should be consistent with the actual application scenario of the neural network. For example, neural networks often need to perform convolution calculations on images of various lengths and widths to obtain convolution results. For example: Even a simple 28*28 length and width handwriting recognition image may require about 784 input ports. Obviously, the size of the storage transistor array is quite large in this case. The more difficult problem at this time is that the length of the word line will become very long as the size of the storage transistor array increases, because the adjustment of the weight depends on the programming of the transistor floating gate to write or erase the charge through the word line. Too long a word line will cause a large word line parasitic capacitance C on the transistor array. P , resulting in errors in weight programming. Parasitic capacitance includes at least the gate-source parasitic capacitance and gate-drain parasitic capacitance of each transistor coupled to the word line and the capacitance of the word line to the substrate. The storage transistor array is prepared on a semiconductor substrate. Programming error also means deviation in weight configuration. It is worth noting that in many neural networks such as convolutional neural networks in intelligent driving domain video perception, the input image resolution related to the number of input ports of the transistor array is usually 1628*1236, 1920*1080, 2048*1024, 3840*2160, etc. Regardless of whether the length and width of the image are scaled, it will be accompanied by a huge array size and give rise to parasitic capacitance C. PThe complexity of the array is that the amount of parasitic capacitance is not completely corresponding to the size of the array. Parasitic capacitance includes both parallel capacitance and series capacitance, such as the body capacitance of the transistor, and coupling capacitance, such as the parasitic coupling capacitance between different word lines. This brings a great error rate to the iteration of weights such as the back-propagation algorithm, and there is great irrationality in both training and reasoning. In conventional application scenarios of memory, there is no need to worry about this problem, because the memory will use segmented physical blocks Blokc or planes or particles LUN to cope with such storage requirements when facing ultra-large or massive data storage applications. However, in the neural network scenario, if the weight matrix is ​​segmented in different physical areas, the inconsistency of the programming conditions of the weight elements will become serious.

[0121] See also Figure 6 In an optional embodiment, for any transistor on any word line of the storage transistor array in the stage of dynamically configuring weights (dynamically configuring weights is because the parameters of the neural network are continuously iterating), before any transistor performs any weight adjustment, the reverse voltage of the programming voltage used by other transistors on the word line adjacent to the previous weight adjustment is first transiently mounted on the word line, and then the programming voltage set for the transistor is applied to the word line to perform the weight adjustment. One of the purposes is to suppress the weight update iteration failure caused by the parasitic charge of the parasitic capacitor existing at the word line of the storage transistor array. The parasitic charge is usually affected by the residual programming voltage of other transistors that implement weight adjustment on the word line. Under high image resolution conditions, assuming that the expected number of iterations of the neural network is epoch=5000, the parasitic charge of any word line will quickly and secretly get out of control under such a large array size and such a large number of iterations. It is difficult to grasp the true value of the actual parasitic charge of the word line, and even if it is known, it will be quickly refreshed by the next round of iteration.

[0122] See also Figure 6 In an optional embodiment, regarding how to better deal with the urgent concerns such as weight offset faced when updating the weights of the neural network inside the memory: the basic solution is to adjust the weight of any transistor in the storage transistor array using its threshold voltage, thereby dynamically configuring the weight matrix of the neural network. A further optimization is that for any transistor on any word line of the storage transistor array in the stage of dynamically configuring the weight, before any weight adjustment is performed, the reverse voltage of the programming voltage used by other transistors on the word line adjacent to the previous weight adjustment is transiently mounted to the word line once, and then the programming voltage set for the transistor is applied to the word line to perform the weight adjustment.

[0123] See also Figure 6For example, the transistor T in a transistor array MN The weight of the threshold voltage V THMN To adjust, transistor T MN It is a transistor determined by the bit line BN and the source line SM and the word line it is on is GM. Set positive integers M = 1, 2, 3..., positive integers N = 1, 2, 3.... This can dynamically configure the weight matrix Ma_wei, V of the neural network NET THMN Medium V TH is the threshold and the subscript MN is the index and it represents the threshold voltage and weight W of any transistor MN All can be manipulated and adjusted via index positions.

[0124] See also Figure 6 For example, any transistor such as T on any word line such as GM of the transistor array MN In the dynamic configuration corresponding weight W MN The stage, any transistor such as T MN Before performing any weight adjustment, for example, before performing the weight adjustment with the number of epochs = 1455, it is preferred to firstly set the transistors on any word line, that is, the word line GM, which are adjacent to the previous weight adjustment (assuming that the transistors T M3 It is T MN The programming voltage used by the transistor that performs the current weight adjustment (e.g., epoch = 1455) adjacent to the previous weight adjustment, for example, V PGM3 The reverse voltage transient is mounted once on any word line GM. After that, the reverse voltage transient is mounted on any transistor such as the transistor T on GM. MN The programming voltage is set as V PGMN is applied to, for example, the word line GM to perform any weight adjustment, for example, the weight W of epoch=1455. MN Suppress the weight (such as W) caused by the parasitic charge of the parasitic capacitance existing at any word line GM of the storage transistor array MN) The update iteration fails and it is worth emphasizing that: this scheme is not suitable for data storage applications in the memory, otherwise it will cause the logical value of the stored data bit to flip unexpectedly with a high probability. Note that the parasitic charge of the word line GM has a volatile characteristic, but the rate of this volatile power loss may be lower than the speed of weight matrix iteration, which is related to the transistor deployment density of the storage transistor array and the selection of components such as wiring and the critical size of the transistor. The above scheme can make up for and save the error rate of the weight as much as possible, and it complements the improvement and training of neural network parameters such as weights. An important goal of the above scheme is also to compress the step difference between the programming voltage set by any transistor to perform any weight adjustment and the programming voltage used by other transistors adjacent to the previous weight adjustment on any word line, so as to avoid the programming voltage set by any transistor to perform any weight adjustment to be coupled to the adjacent word line of any word line. Thereby preventing the weight values ​​of some transistors on adjacent word lines from being changed in unexpected iterations.

[0125] See also Figure 6 , for example, the above scheme tries to compress any transistor such as T MN The programming voltage set for any weight adjustment is V PGMN Other transistors adjacent to any word line such as GM in the previous weight adjustment, such as transistor T M3 The programming voltage used is V PGM3 The voltage step between them is to avoid any transistor, such as transistor T MN Execute any one time (for example, the execution number is epoch=1455) to adjust the weight setting programming voltage, such as voltage V PGMN Coupled to the adjacent word line GM-1 or GM+1 of any word line GM. In the memory transistor array, if the voltage V PGMN When coupled to positions such as GM-1 or GM+1, the weight values ​​of some transistors coupled to word lines GM-1 or GM+1 (such as the intersection of GM-1 / GM+1 and BN) will be inadvertently changed in a non-iterative manner and the weight matrix will be modified locally rather than globally. In this case, the neural network will produce wrong results whether it is training or reasoning. The above-mentioned solution designed by the present application cleverly resolves the problem of undesired iterative changes in the weight values ​​of some transistors on adjacent word lines.

[0126] See also Figure 6Based on the previous text, the convolutional neural network is trained with training samples, and the trained parameters such as weight matrix have good generalization performance for the test samples. It has considerable robustness and recovery of non-accidental offsets in resisting the wrong weights caused by parasitic capacitance on any word line in the storage transistor array. It is a convolutional neural network or sub-network with good performance. Neural networks are typically suitable for intelligent voice models, end-side image reasoning models, and most other AI artificial intelligence scenarios.

[0127] The above description and drawings provide typical embodiments of specific structures of specific embodiments. The above invention provides existing preferred embodiments, but these contents are not intended to be limiting. For those skilled in the art, various changes and modifications will undoubtedly be obvious after reading the above description. Therefore, the attached claims should be regarded as covering all changes and modifications of the true intent and scope of the present invention. Any and all equivalent scopes and contents within the scope of the claims should be considered to still be within the intent and scope of the present invention.

Claims

1. A method for performing in-memory computing, characterized in that: include: A storage transistor array is used in a flash memory as a carrier of a weight matrix of a neural network. The input data of the neural network is set to be transmitted to a series of bit lines of the storage transistor array, and the output data of the neural network is set to be captured from a series of source lines of the storage transistor array. The weight of any transistor in the storage transistor array is adjusted using its threshold voltage, thereby dynamically configuring the weight matrix of the neural network.

2. The method according to claim 1, characterized in that: The memory transistor array for the neural network is located on the same physical block of flash memory; and The storage space where the input data and output data of the neural network are located and the storage space where the storage transistor array is located belong to different physical blocks of the flash memory respectively.

3. The method according to claim 1, characterized in that: The raw input data of the neural network is converted into input quantities in the form of voltage values, which are transmitted to a series of bit lines of the memory transistor array based on the input quantities sorted into vectors.

4. The method according to claim 1, characterized in that: The output data of the neural network is represented by the output current of the storage transistor array, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

5. The method according to claim 1, characterized in that: The method of adjusting the weight of any transistor includes changing the floating gate injection charge, and adjusting the weight of any transistor at its position in the memory transistor array by changing the floating gate injection charge of the transistor.

6. The method according to claim 5, characterized in that: During the back-propagation phase of the neural network, the weight matrix of the neural network is dynamically configured by resetting the weights at various locations in the storage transistor array.

7. The method according to claim 3, characterized in that: A digital-to-analog converter is configured at each bit line of the storage transistor array, and is used to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line.

8. The method according to claim 4, characterized in that: An analog-to-digital converter is configured at each source line of the storage transistor array, and is used to convert an output signal in an output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

9. A device for performing in-memory computing, characterized in that: include: A flash memory, wherein the memory transistor array is used as a carrier of the weight matrix of the neural network, and the input data of the neural network is set to be transmitted to a series of bit lines of the memory transistor array, and the output data of the neural network is set to be captured from a series of source lines of the memory transistor array; An input unit that converts raw input data of the neural network into an input quantity in the form of a voltage value to be transmitted to a series of bit lines of the storage transistor array based on the input quantity sorted into a vector; An output unit is used to convert an output signal in the output current format at the source line into output data compatible with the digital output quantity of the neural network, and the output current of a series of source lines of the storage transistor array is regarded as the output quantity of the storage transistor array sorted into a vector.

10. A storage system for performing in-memory computing, characterized in that: include: A memory chip based on a flash memory, in which a memory transistor array is used as a carrier of a weight matrix of a neural network, wherein the weight of any transistor is adjusted by its threshold voltage, thereby dynamically configuring the weight matrix of the neural network; setting the input data of the neural network to be transmitted to a series of bit lines of the memory transistor array and also setting the output data of the neural network to be captured from a series of source lines of the memory transistor array; A digital-to-analog converter built into the memory chip, wherein a digital-to-analog converter is configured at each bit line of the memory transistor array to convert the input data of the neural network into a voltage value that matches the input signal pattern of the bit line; An analog-to-digital converter is built into the memory chip, wherein an analog-to-digital converter is configured at each source line of the memory transistor array to convert an output signal in the output current format at the source line into output data that is compatible with the digital output quantity of the neural network.

Images