Memory device and operation method thereof

Abstract

The present invention discloses a memory device and an operation method thereof, wherein the memory device is used to implement an artificial neural network, and the operation method comprises: reading out a weight matrix of a current layer among multiple layers of the artificial neural network from the memory device to extract a plurality of neural values; determining whether to perform correction; when determining that correction is to be performed, recalculating and updating an average value and a variation value of the neural values; and performing batch normalization using the average value and the variation value of the neural values.

Description

Technical Field

[0001] The invention relates to a memory device and an operating method thereof. Background Art

[0002] In a non-volatile memory (NVM) based neuromorphic system, memory cells can be used to store weights.

[0003] Figure 1 A schematic diagram showing a system of an existing artificial neural network (ANN). Figure 1 As shown, the artificial neural network 100 includes a plurality of convolution layers 110-1 to 110-6. In the present specification, "convolution layer" and "layer" have the same meaning. In the process of training the artificial neural network, a computer can be used to adjust or train the weights. After the adjustment or training is completed, the computer writes the weights into the memory unit.

[0004] In artificial neural networks, there is a weight matrix (WM) between layers. The input of a layer multiplied by the weight matrix can get the output of the layer, which is then output to the next layer. In addition, activation functions (AF) can be performed between layers. After the last convolutional layer is operated, the fully connected layer is used to perform operations to obtain the output.

[0005] If necessary, the weights stored in the memory cells can be modified by the computer. In general, if non-volatile memory is used to implement artificial neural networks, problems such as weight fluctuations, program errors, and retention loss may occur, resulting in a decrease in the accuracy of the artificial neural network. Weight fluctuation means that although the actual weight in the memory cell is 0.2, the weight read out each time may change (for example, the weight read out last time was 0.2, but the weight read out currently may be 0.1). Programming error means that the ideal weight obtained through computer simulation or training is 0.2 (that is, the ideal weight originally expected to be written into the memory cell is 0.2), but after programming, the actual weight in the memory cell may become 0.3. Retention loss means that after long-term use, the memory state of the memory cell used to store the weight changes, resulting in the loss of the content stored in the memory cell, so that incorrect weights may be read out.

[0006] In addition, analysis shows that if non-volatile memory is used in the early layers, weight jumps, programming errors, and retention losses will significantly reduce the accuracy.

[0007] Therefore, a memory device and an operating method thereof are needed, which can effectively improve the problem of decreased accuracy of the artificial neural network even if a non-volatile memory is used to implement the artificial neural network. Summary of the invention

[0008] According to an exemplary embodiment of the present case, a method for operating a memory device is proposed, wherein the memory device is used to implement an artificial neural network, and the method includes: reading out a weight matrix of a current layer among multiple layers of the artificial neural network from the memory device to retrieve a plurality of neural values; determining whether to perform correction; when it is determined that correction is to be performed, recalculating and updating an average value and a variance value of the neural values; and performing batch normalization using the average value and the variance value of the neural values.

[0009] According to an exemplary embodiment of the present case, a memory device is proposed for implementing an artificial neural network, the memory device comprising: a memory array for storing individual weight matrices of multiple layers of the artificial neural network; and a controller coupled to the memory array, the controller structure being configured to: read out a weight matrix of a current layer of the artificial neural network from the memory array to retrieve a plurality of neural values; determine whether to perform correction; when it is determined that correction is to be performed, recalculate and update an average value and a variance value of these neural values; and perform batch normalization using the average value and the variance value of these neural values.

[0010] In order to better understand the above and other aspects of the present invention, embodiments are given below and described in detail with reference to the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A system diagram showing a conventional artificial neural network.

[0012] Figure 2 A functional diagram of a memory device according to an exemplary embodiment of the present invention is shown.

[0013] Figure 3 A flow chart of a memory operation method according to an exemplary embodiment of the present invention is shown.

[0014] Figure 4 A data flow diagram according to an exemplary embodiment of the present invention is shown.

[0015]

Explanation of symbols

[0016] 100: Artificial Neural Network 110-1~110-6: Convolutional Layer

[0017] WM: weight matrix AF: activation function

[0018] 200: memory device 210: memory array

[0019] 220: Controller

[0020] 310-350: Steps

[0021] V1, V2 and V3: input of the current layer

[0022] G11~G33: Weight Xi': Neural value

[0023] Current: I11~I33 DETAILED DESCRIPTION

[0024] The technical terms used in this specification refer to the customary terms in the technical field. If some terms are explained or defined in this specification, the interpretation of these terms shall be based on the explanation or definition in this specification. Each embodiment of the present disclosure has one or more technical features. Under the premise of possible implementation, a person with ordinary knowledge in the technical field can selectively implement some or all of the technical features in any embodiment, or selectively combine some or all of the technical features in these embodiments.

[0025] Figure 2 FIG. 4 is a functional diagram of a memory device according to an exemplary embodiment of the present invention. Figure 2As shown, the memory device 200 includes a memory array 210 and a controller 220. At least one weight matrix (WM) of the artificial neural network can be stored in the memory array 210. The controller 220 is coupled to the memory array 210 to control the operation of the memory array 210 to implement the artificial neural network.

[0026] Figure 3 A flow chart of a memory operation method according to an exemplary embodiment of the present invention is shown. Figure 3 As shown, in step 310, a weight matrix of a current layer in the multiple layers of the artificial neural network is read out from the memory device to retrieve the neuron value. In step 320, it is determined whether to perform correction of the mean value and the variation value of the neuron value. The timing of correcting the mean value and the variation value of the neuron value is, for example but not limited to, when the new weight is first written to the memory array, or when the weight drift is caused by long-term retention, or when it is triggered by the user. For example, when the user feels that the accuracy of the artificial neural network has decreased, the user can trigger the correction of the mean value and the variation value of the neuron value.

[0027] If step 320 is yes, then in step 330, the mean and variance of the neural values ​​are recalculated and updated. After step 330 is completed, step 340 is continued. If step 320 is no, step 340 is executed. In step 340, batch normalization is performed using the mean and variance of the neural values. In step 350, the activation function is executed. In the exemplary embodiment of the present case, steps 310-350 are executed by the controller 220, for example.

[0028] Figure 4 A schematic diagram of data flow according to an exemplary embodiment of the present invention is shown. Figure 4 In the figure, the weight matrix of the current layer is a 3*3 weight matrix (including weights G11-G33), but it should be understood that it is used for illustration and the case is not limited to this.

[0029] like Figure 4As shown, the output of the previous layer is V1, V2 and V3, which are used as the input of the current layer. Input V1, V2 and V3 are multiplied by the weight matrix to obtain I11=V1*G11; I12=V2*G12; I13=V3*G13; I21=V1*G21; I22=V2*G22; I23=V3*G23; I31=V1*G31; I32=V2*G32; I33=V3*G33. I11~I33 represent currents, which are used to form the neural value Xi'. Neural value Xi'=∑I. For example, X1'=I11+I12+I13; X2'=I21+I22+I23; X3'=I31+I32+I33.

[0030] Now, how to obtain the average value of the neural value in the exemplary embodiment of the present case will be described. This is for illustrative purposes only and the present case is not limited thereto. The average value of the neural value can be obtained, for example, by the following equation (1):

[0031]

[0032] Now, how to obtain the variation value of the neural value in the exemplary embodiment of this case will be described. This is for illustrative purposes only and this case is not limited thereto. The variation value of the neural value can be obtained, for example, by the following equation (2):

[0033]

[0034] Now, how to perform batch normalization in the exemplary embodiment of the present case is described. This is for illustrative purposes only and the present case is not limited thereto. Batch normalization can be obtained, for example, by the following equations (3) and (4):

[0035]

[0036]

[0037] In equation (3), ∈ is the error term, γ is the scale parameter (which can be defined by the user), and β is the shift parameter (which can be defined by the user).

[0038] In the exemplary embodiment of the present case, the activation function includes, but is not limited to, a rectified linear unit (ReLU), a max pooling method, a drop out, etc. This is for illustration only, and the present case is not limited thereto.

[0039] That is to say, in this case Figure 3In the flowchart of , when it is determined in step 320 that the mean value and the variance value of the neural value need to be corrected (that is, step 320 is yes), before batch normalization is performed, the weights are actually read from the memory array, and the neural values ​​(the neural values ​​may include weight errors) are obtained to recalculate and update the mean value and the variance value of the neural value, and batch normalization is performed. In addition, if it is determined in step 320 that the mean value and the variance value of the neural value do not need to be corrected (that is, step 320 is no), the mean value and the variance value of the neural value obtained previously can be used to perform batch normalization.

[0040] In other possible embodiments of the present invention, the activation function may be executed before "recalculating and updating the mean and variance of the neural values", which is also within the spirit of the present invention.

[0041] In the prior art, when the weights stored in the non-volatile memory array have errors due to various factors (such as weight jump, programming error or retention loss), the prior art still uses ideal weights to obtain neural values, so its accuracy is easily affected by weight errors.

[0042] In the exemplary embodiment of the present case, in order to improve the accuracy of the artificial neural network, the following features are performed: (1) extracting the neural values ​​in each layer of the artificial neural network; (2) calculating the average value and variance value of the neural values ​​in each layer; (3) updating the average value and variance value of the neural values ​​in each layer. That is, in the present embodiment, even if the artificial neural network is implemented using a non-volatile memory, the accuracy can still be improved. This is because, when the weights stored in the non-volatile memory array have errors due to various factors, in the present embodiment, the actual weights are read from the memory array, and the average value and variance value of the neural values ​​are recalculated and updated accordingly. Therefore, the recalculated average value and variance value of the neural value have taken the weight error into consideration, so the accuracy can be improved.

[0043] In addition, in the present embodiment, the reason for extracting the neural value from the artificial neural network is that: (1) Since weight jumps, programming errors or retention losses will reduce the reliability of the memory, the neural value obtained by the sum-of-product operation will cause errors, and this error will be transmitted throughout the artificial neural network, resulting in a serious decrease in accuracy. (2) The weight error caused by programming errors and weight jumps can be monitored through the distribution of neural values. Even long-term retention losses may cause weight drift and neural value errors. By updating the new parameters of the neural value (i.e., the new mean and the new variance) according to the distribution of neural values, the accuracy loss can be repaired.

[0044] In addition, in the present embodiment, after recalculating the average and variance of the neural values, the updated average and variance of the neural values ​​can be written back to the memory array to complete the repair procedure, thereby eliminating the need for a re-programming procedure and reducing complexity.

[0045] In the present embodiment, the artificial neural network can be implemented in any type of memory (such as static random access memory (SRAM), dynamic random access memory (DRAM), NVM, embedded dynamic random access memory (eDRAM), embedded non-volatile memory (eNVM), magnetoresistive random access memory (MRAM), phase change memory (PCM), resistive random-access memory (RRAM or ReRAM), ferroelectric random access memory (FeRAM or FRAM), etc.).

[0046] The accuracy improvement method of the present embodiment can be applied to any type of artificial neural network. In addition, the accuracy improvement method of the present embodiment can also be used to handle any type of error in the memory array (such as but not limited to programming error, read fluctuation, read disturb, etc.). The present embodiment can be applied to any system that is susceptible to errors and needs to repair these errors.

[0047] In summary, although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for operating a memory device, the memory device being used to implement an artificial neural network, the method comprising: Reading a weight matrix of a current layer of the artificial neural network from the memory device to retrieve a plurality of neural values, the plurality of neural values ​​being composed of electric currents; Determine whether to correct the mean and variance of the neural values; When it is determined that correction is required, the mean and variance values ​​of these neural values ​​are recalculated and updated; as well as Batch normalization is performed using the mean and variance of the neural values; The determination of whether to correct the mean value and variation value of the neural value includes: When a weight is first written to the memory device, determining that a correction is required; or When a weight drift occurs, it is determined that a correction is required; or When triggered by a user, it is determined that calibration is required; Recalculating and updating the mean and variance of these neural values ​​includes: when it is determined that the mean and variance of the neural values ​​need to be corrected, before batch normalization is performed, the weights are actually read out from the memory array, and the neural values ​​are obtained based on them to recalculate and update the mean and variance of the neural values, and batch normalization is performed.

2. The operating method according to claim 1, wherein: After performing batch normalization, an activation function is performed.

3. The operating method according to claim 1, wherein: Before recalculating and updating the mean value and the variance value of the neural values, an activation function is executed.

4. A memory device for implementing an artificial neural network, the memory device comprising: a memory array for storing individual weight matrices of multiple layers of the artificial neural network; as well as a controller coupled to the memory array, The controller architecture consists of: Reading a weight matrix of a current layer of the artificial neural network from the memory array to retrieve a plurality of neural values, the plurality of neural values ​​being composed of electric currents; Determine whether to correct the mean and variance of the neural values; When it is determined that correction is required, the mean and variance values ​​of these neural values ​​are recalculated and updated; as well as Batch normalization is performed using the mean and variance of the neural values; The determination of whether to correct the mean value and variation value of the neural value includes: When a weight is first written to the memory device, determining that a correction is required; or When a weight drift occurs, it is determined that a correction is required; or When triggered by a user, it is determined that calibration is required; Recalculating and updating the mean and variance of these neural values ​​includes: when it is determined that the mean and variance of the neural values ​​need to be corrected, before batch normalization is performed, the weights are actually read out from the memory array, and the neural values ​​are obtained based on them to recalculate and update the mean and variance of the neural values, and batch normalization is performed.

5. The memory device of claim 4, wherein: The controller architecture is configured to execute an activation function after executing batch normalization.

6. The memory device of claim 4, wherein: The controller is structured to execute an activation function before recalculating and updating the average value and the variance value of the neural values.

Images