On-chip fine tuning method and device on in-memory computing chip based on group-by-group fine tuning

Abstract

The invention discloses an on-chip fine tuning method and device on a memory computing chip based on group-by-group fine tuning, and the method comprises the steps: firstly grouping a plurality of computing layers in a neural network when the neural network is trained, and calculating the standard output corresponding to each group of computing layers based on the grouping condition and calibration data; then, actual output of a group where the calculation layer receiving the calibration data set is located is calculated through the calibration data set and the weight value of the calculation layer receiving the calibration data set; and finally, the difference between the actual output of the group where the calculation layer of the received calibration data set is located and the corresponding standard output is calculated according to the loss function, and the weight value in the neural network is adjusted according to the difference. According to the technical scheme, the use of a memory and the demand of a calibration data set are reduced, in addition, the complexity and the power consumption of hardware are reduced, meanwhile, calculation delay brought by a calibrator module is avoided, large-scale calibration data sets are not needed, and overfitting is avoided.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to an on-chip fine-tuning method and device on an in-memory computing chip based on group-by-group fine-tuning. Background Art

[0002] Compute In Memory (CIM) is an emerging computing architecture that is widely used in artificial intelligence (AI) chips and neural network accelerators. As the scale of neural network algorithms becomes increasingly large, the amount of computational data and weight parameters involved increases dramatically (especially in data-intensive tasks). In order to cope with the sharp increase in computational data and weight parameters, in-memory computing (computing directly at the location of stored data) is proposed, that is, integrating computing and storage functions together, eliminating a large amount of unnecessary data transmission, thereby significantly improving computing efficiency and greatly reducing energy consumption.

[0003] However, when the traditional in-memory computing array structure performs in-memory computing, the calculation results are easily affected by factors such as process deviation, temperature fluctuation, voltage drift and noise interference. In order to solve this influence, on-chip adjustment technology is used to enhance the accuracy of the neural network. Figure 2 An on-chip tuning method to enhance the accuracy of neural networks is shown in FIG. Figure 2 The method shown in the figure first performs forward propagation to obtain the final output of the network when performing on-chip adjustment. Then, a loss function is constructed based on the difference between the predicted output and the final output of the neural network to quantify the error. Through the back-propagation algorithm, the gradient of the loss function is used to update the network weights to reduce the prediction error and improve the accuracy of the neural network. Figure 3 Another on-chip tuning method to enhance the accuracy of neural networks is shown in Figure 2. Figure 3 When the method shown in the figure performs on-chip adjustment, a calibrator module is introduced, and the calibrator module is added to the computing layer, and the accuracy of the neural network is improved by the calibrator module. Although the on-chip adjustment method shown in the figure can improve the accuracy of the neural network, it requires a complete training cycle for the entire neural network when applied, that is, the entire data set must be accessed, and the end-to-end training process requires the storage of the output data of each layer in the network for subsequent back propagation, which significantly increases the memory usage and the demand for the size of the data set. The on-chip adjustment method shown in the figure can also improve the accuracy of the neural network, but it requires the addition of a calibrator module when it is applied, which increases the complexity and power consumption of the hardware, and the calibrator module and the computing layer are serial operations, which further increases the calculation delay, and the calibrator module needs to be trained, and a large-scale calibration data set is also required to avoid overfitting.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide an on-chip fine-tuning method and device on an in-memory computing chip based on group-by-group fine-tuning, so as to solve the problem that although the on-chip adjustment method in the prior art can improve the accuracy of the neural network, its end-to-end training process when applied needs to store the output data of each layer in the network for subsequent back propagation, which significantly increases the demand for memory usage and data set size, or, the use of a calibrator module increases the complexity and power consumption of the hardware, and the calibrator module and the computing layer are serial operations, which further increases the calculation delay, and the calibrator module needs to be trained, and a large-scale calibration data set is also required to avoid overfitting problems.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide an on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning, which is used to train a neural network, wherein the neural network includes a plurality of computing layers connected in sequence, wherein the plurality of computing layers include a convolutional layer and a fully connected layer which is the output end of the neural network, and the on-chip fine-tuning method includes:

[0007] Step A, grouping a plurality of computing layers in the neural network, wherein the plurality of computing layers are divided into at least two groups;

[0008] Step B, based on the grouping of the plurality of sequentially connected computing layers and the calibration data set, calculating the standard output corresponding to each group of computing layers;

[0009] Step C, receiving a calibration data set;

[0010] Step D, calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set;

[0011] Step E, calculating the difference between the actual output of the group of computing layers that received the calibration data set and the corresponding standard output according to the loss function;

[0012] When the gap meets the requirement, the weight value corresponding to the computing layer that meets the requirement and receives the calibration data set is frozen, and the actual output of the group of computing layers that meet the requirement and receive the calibration data set is input as a new set of calibration data sets to the next connected computing layer, and the next connected computing layer is executed from step D;

[0013] When the gap does not meet the requirements, a gradient descent algorithm is used to calculate the adjustment increment of the weight value of the calculation layer that receives the calibration data set, and the weight value of the calculation layer that receives the calibration data set is adjusted according to the adjustment increment of the weight value;

[0014] After the weight value is adjusted, the process starts from step D until the gap meets the requirement;

[0015] The calibration data set is input from the computing layer at the starting position in the neural network.

[0016] In a further configuration of the present invention, before calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set, the method comprises:

[0017] Pre-train the weight values ​​of several computing layers, and deploy the weight values ​​on the in-memory computing chip.

[0018] A further configuration of the present invention is that when the gap does not meet the requirements, a gradient descent algorithm is used to calculate an adjustment increment of a weight value of a calculation layer of the calibration data set received for which the gap does not meet the requirements, and the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirements is adjusted according to the adjustment increment of the weight value, and after the weight value is adjusted, the step D is started to be executed until the gap meets the requirements, including:

[0019] Get the single adjustment increment when adjusting the weight value;

[0020] After obtaining n adjustment increments, the adjustment increments of the adjusted weight values ​​obtained n times are added to the weight value of the calculation layer of the calibration data set received when the gap does not meet the requirements to adjust the weight value, and after the weight value is adjusted, execution is started from the step D until the gap meets the requirements, where n is a positive integer.

[0021] According to a further configuration of the present invention, the standard output is an ideal output result corresponding to the computing layer of each group in the neural network, which is pre-calculated based on the grouping of several sequentially connected computing layers using a digital computing module, a trained neural network and a calibration data set.

[0022] According to a further configuration of the present invention, the loss function includes a mean square error.

[0023] A further configuration of the present invention is that after the weight values ​​of the computing layers of each group in the neural network are adjusted, the neural network is inferred using the adjusted weight values.

[0024] The present invention also provides an on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning, which is used to train a neural network, wherein the neural network includes a plurality of sequentially connected computing layers, wherein the plurality of computing layers include a convolutional layer and a fully connected layer which is an output end of the neural network, and the on-chip fine-tuning device includes:

[0025] A neural network grouping module, used for grouping a plurality of computing layers in the neural network on the in-memory computing chip, wherein the plurality of computing layers are divided into at least two groups;

[0026] A digital calculation module, used for calculating the standard output corresponding to each group of the calculation layers based on the grouping conditions of the plurality of sequentially connected calculation layers and the calibration data set;

[0027] An in-memory computing chip for receiving a calibration data set;

[0028] The in-memory computing chip is also used to perform step D in the fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning as claimed in any one of claims 1 to 6;

[0029] The digital computing module is also used to perform step E in the fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning as claimed in any one of claims 1 to 6;

[0030] The in-memory computing chip is also used to freeze the weight value corresponding to the computing layer that receives the calibration data set when the gap meets the requirement, and input the actual output of the group of computing layers that receive the calibration data set as a new calibration data set to the next connected computing layer, and execute the next connected computing layer from step D.

[0031] The digital calculation module is also used to use a gradient descent algorithm to calculate the adjustment increment of the weight value of the calculation layer of the calibration data set received by the gap not meeting the requirement when the gap does not meet the requirement, and adjust the weight value of the calculation layer of the calibration data set received by the gap not meeting the requirement according to the adjustment increment of the weight value;

[0032] After the weight value is adjusted, the process starts from step D until the gap meets the requirement;

[0033] The calibration data set is input from the computing layer at the starting position in the neural network.

[0034] According to a further configuration of the present invention, the on-chip fine-tuning device further comprises:

[0035] The digital computing module is used to pre-train the weight values ​​of several computing layers and deploy the weight values ​​on the in-memory computing chip.

[0036] According to a further configuration of the present invention, the on-chip fine-tuning device further comprises:

[0037] A digital assisted fine-tuning module is used to obtain a single adjustment increment when adjusting the weight value;

[0038] It is also used to accumulate the adjustment increments when adjusting the weight values ​​obtained n times to the weight value of the calculation layer of the calibration data set received when the gap does not meet the requirements after obtaining n adjustment increments to adjust the weight value, and start executing from the step D after the weight value is adjusted until the gap meets the requirements, where n is a positive integer.

[0039] According to a further configuration of the present invention, the digital computing module is a graphics processor.

[0040] The beneficial effects of the present invention are:

[0041] Compared with the on-chip adjustment method in the prior art, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning of the present invention is as follows: when training a neural network, several computing layers in the neural network are first grouped, and the standard output corresponding to the computing layer of each group is calculated based on the grouping situation and the calibration data; then, the actual output of the group to which the computing layer receiving the calibration data set belongs is calculated by the calibration data set and the weight value of the computing layer receiving the calibration data set; finally, the gap between the actual output of the group to which the computing layer receiving the calibration data set belongs and the corresponding standard output is calculated according to the loss function, and when the gap meets the requirement, the weight value corresponding to the computing layer receiving the calibration data set that meets the requirement is frozen, and the actual output of the group to which the computing layer receiving the calibration data set that meets the requirement is taken as a A new calibration data set is input to the next group of connected computing layers, and the weight values ​​of the next group of connected computing layers are adjusted. When the gap does not meet the requirements, the gradient descent algorithm is used to calculate the adjustment increment of the weight value of the computing layer that receives the calibration data set and the gap does not meet the requirements. According to the adjustment increment of the weight value, the weight value of the computing layer that receives the calibration data set and the gap does not meet the requirements is adjusted. After the weight value is adjusted, the gap between the actual output of the computing layer that receives the calibration data set and the corresponding standard output is further calculated based on the adjusted weight value to see whether it meets the requirements. If yes, the weight value of the next group of connected computing layers is adjusted. If not, the weight value of the computing layer that receives the calibration data set and the gap does not meet the requirements continues to be adjusted until the gap meets the requirements. It can be seen that the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning adjusts the weight value in a grouped manner. When the gap between the actual output and the standard output of a group of computing layers does not meet the requirements, only the weight value corresponding to the gap between the actual output and the standard output of the group of computing layers that does not meet the requirements is adjusted. Only a small amount of calibration data set is required, which reduces the use of memory and the demand for calibration data sets. In addition, there is no need to add a calibrator module, which reduces the complexity and power consumption of the hardware. At the same time, it avoids the calculation delay caused by the calibrator module and the need for a large-scale calibration data set to avoid overfitting. Because the calibration data set is input from the computing layer at the starting position in the neural network, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning can complete the training of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary personnel in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0043] Figure 1 It is a cross-array in-memory computing array structure in the prior art.

[0044] Figure 2 It is an end-to-end neural network training structure diagram in the prior art.

[0045] Figure 3 It is a diagram of the neural network training structure using a calibrator module in the prior art.

[0046] Figure 4 It is a flow chart of an on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning of the present invention.

[0047] Figure 5 The present invention is a flowchart of adjusting weight values ​​in an embodiment.

[0048] Figure 6 It is a principle block diagram of an on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning of the present invention.

[0049] Figure 7 The figure is a block diagram of the connection between the digital computing module and the in-memory computing chip in one embodiment.

[0050] Figure 8 is a block diagram of a neural network having three computing layers in one embodiment.

[0051] Fig. 9 FIG. 1 is a diagram showing the grouping of a neural network having three computing layers in one embodiment.

[0052] Fig.10 The figure is a principle block diagram of adjusting the weight value of the convolution layer 1 in an embodiment.

[0053] Fig.11 The figure is a principle block diagram of adjusting the weight value of the convolutional layer 2 in an embodiment.

[0054] Fig.12 The figure is a block diagram of the principle of adjusting the weight value of the fully connected layer in an embodiment.

[0055] Fig.13 is a diagram showing the grouping of a neural network with three computing layers in another embodiment.

[0056] Fig.14 The figure is a principle block diagram of the weight value adjusted by the digital auxiliary fine-tuning module in one embodiment.

[0057] Fig.15 The figure is a block diagram of the principle of reasoning on a neural network in one embodiment.

[0058] The marks in the accompanying drawings are: 10, neural network grouping module; 20, in-memory computing chip; 30, digital computing module; 40, digital fine-tuning auxiliary module. DETAILED DESCRIPTION

[0059] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described in detail with reference to the accompanying drawings. In the following description, it should be understood that the directions or positional relationships indicated by "front", "back", "up", "down", "left", "right", "longitudinal", "horizontal", "vertical", "horizontal", "top", "bottom", "inside", "outside", "head", "tail", etc. are based on the directions or positional relationships shown in the accompanying drawings, are constructed and operated in a specific direction, and are only for the convenience of describing the present technical solution, rather than indicating that the device or element referred to must have a specific direction, and therefore cannot be understood as a limitation to the present invention.

[0060] Compute In Memory (CIM) is an emerging computing architecture that is widely used in artificial intelligence (AI) chips and neural network accelerators. As the scale of neural network algorithms becomes increasingly large, the amount of computational data and weight parameters involved increases dramatically (especially in data-intensive tasks). The traditional von Neumann architecture separates the computing unit from the storage unit, resulting in frequent data transmission between the two. This not only limits the computing speed, but also consumes a lot of energy due to data transmission and memory read and write operations, and its energy consumption often exceeds the energy consumption of the computing unit itself. In order to solve this problem, in-memory computing (directly calculating through the location of the stored data) is proposed, that is, integrating computing and storage functions together, eliminating a large amount of unnecessary data transmission, thereby significantly improving computing efficiency and greatly reducing energy consumption.

[0061] At present, in-memory computing mainly uses a crossbar array built on a storage array to perform multiplication and addition operations, such as Figure 1 As shown, a cross-array in-memory computing array structure in the prior art is shown. The cross-array in-memory computing array structure can use analog signals to perform multiplication and addition operations, rather than relying on traditional digital multipliers and adders, wherein the analog multiplication is simulated by Ohm's law, and the analog addition is performed based on Kirchhoff's voltage / current law or the series-parallel characteristics of resistors. The in-memory computing method under the cross-array in-memory computing array structure has the advantages of high throughput, small size and low power consumption.

[0062] However, the in-memory computing method under the cross-array in-memory computing array structure also has its shortcomings. The circuit used to implement the cross-array in-memory computing array structure is more sensitive than the traditional digital computing circuit, and its calculation results are easily affected by factors such as process deviation, temperature fluctuation, voltage drift and noise interference. In addition, the storage value of the analog storage medium may change with time and the increase in the number of uses, which may further cause the trained weights on the graphics processor to be migrated to the analog in-memory computing platform. Even if the input conditions are the same, different results may be obtained, which will affect the accuracy of the calculation. Therefore, the reliability and accuracy of the in-memory computing need to be improved.

[0063] In order to overcome the negative impact of noise interference, process deviation and writing error on neural network performance that may be encountered after analog in-memory computing is deployed on-chip, on-chip adjustment technology is used to enhance the accuracy of neural networks.

[0064] like Figure 2 As shown, an on-chip adjustment method is used to enhance the accuracy of a neural network, wherein, during the on-chip adjustment, forward propagation is first performed to obtain the final output of the network, and then a loss function is constructed based on the difference between the predicted output and the final output of the neural network to quantify the error. Through the back-propagation algorithm, the gradient of the loss function is used to update the network weights to reduce the prediction error and improve the accuracy of the neural network.

[0065] Specifically, for the i-th layer of a neural network, w i represents the parameters of the i-th layer, a i and a i+1 are the input and output activation values ​​of the i-th layer, L is the loss function, and during the back propagation of , given the loss function for a i+1 The derivative of the i-th layer has two goals: to calculate the loss function for a i The derivative of the loss function with respect to w i Without loss of generality, assuming that the i-th layer is a linear layer, its forward propagation process is defined as: i+1 =a i w+b , when the batch size is 1, the back propagation process is determined by the following calculation formula:

[0066]

[0067] This on-chip adjustment method requires a complete training cycle for the entire network, which must access the entire dataset. The end-to-end training process needs to store the output data of each layer in the network for subsequent backpropagation, which significantly increases the memory usage and dataset size requirements.

[0068] like Figure 3As shown, another on-chip adjustment method is used to improve the accuracy of the neural network through a calibrator module. Figure 3 Although the on-chip adjustment method shown in the figure can improve the accuracy of the neural network, its application requires a complete training cycle for the entire neural network, that is, the entire data set must be accessed, and the end-to-end training process requires the storage of the output data of each layer in the network for subsequent back propagation, which significantly increases the memory usage and the data set size requirements. The on-chip adjustment method shown in the figure can also improve the accuracy of the neural network, but its application requires the addition of a calibrator module, which increases the complexity and power consumption of the hardware, and the calibrator module and the computing layer are serial operations, which further increases the computing delay. At the same time, if there is a large deviation during the writing process, the calibrator module may not be able to effectively correct it, and the calibrator module needs to be trained, and a large-scale calibration data set is also required to avoid overfitting.

[0069] Based on this, Figure 6 , Figure 7 As shown, the present invention provides an on-chip fine-tuning device on an in-memory computing chip 20 based on group-by-group fine-tuning, and the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning is used to train a neural network, wherein the neural network includes a plurality of computing layers connected in sequence, and the plurality of computing layers include a convolutional layer and a fully connected layer which is the output end of the neural network.

[0070] The on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning may include a neural network grouping module 10, a digital computing module 30 and an in-memory computing chip 20.

[0071] The neural network grouping module 10 is used to group a number of computing layers in the neural network on the in-memory computing chip 20, and the number of computing layers are divided into at least two groups.

[0072] The digital calculation module 30 is used to calculate the standard output corresponding to each group of the calculation layers based on the grouping of a plurality of sequentially connected calculation layers and the calibration data set.

[0073] The in-memory computing chip 20 is used to receive the calibration data set; the in-memory computing chip 20 is also used to perform the following steps:

[0074] S140, calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set;

[0075] The digital calculation module 30 is also used to perform the following steps:

[0076] S150, calculating the difference between the actual output of the group of the computing layer that receives the calibration data set and the corresponding standard output according to the loss function;

[0077] The in-memory computing chip 20 is further used to freeze the weight value corresponding to the computing layer that receives the calibration data set when the gap meets the requirement, and input the actual output of the group of computing layers that receive the calibration data set as a new calibration data set to the next connected computing layer, and execute the next connected computing layer from S140.

[0078] The digital calculation module 30 is also used to, when the gap does not meet the requirements, use a gradient descent algorithm to calculate the adjustment increment of the weight value of the calculation layer of the calibration data set received by the gap does not meet the requirements, and adjust the weight value of the calculation layer of the calibration data set received by the gap does not meet the requirements according to the adjustment increment of the weight value;

[0079] After the weight value is adjusted, the above S140 is executed until the gap meets the requirement;

[0080] Among them, the calibration dataset is input from the computing layer at the starting position in the neural network.

[0081] In this embodiment, when the neural network is layered, those skilled in the art may group according to actual needs when the number of grouping layers is at least two. For example, one computing layer may be grouped into one group, two sequentially connected computing layers may be grouped into one group, or three sequentially connected computing layers may be grouped into one group.

[0082] Here, it should be noted that when grouping neural networks, when the neural network contains more computing layers, it is necessary to consider the problem of gradient vanishing or gradient exploding. Therefore, additional regularization is needed to alleviate it.

[0083] For example, Figure 8 As shown in , it is a neural network with three computing layers. Fig. 9 As shown, when grouping, each computing layer is a group (convolutional layer 1 in the figure is a group, convolutional layer 2 is a group, and the fully connected layer is a group), that is, the neural network with three computing layers has three groups after grouping.

[0084] For example, Fig.13 As shown, when layering, the computational layers at the starting end of the neural network are grouped as one group (convolutional layer 1 in the figure is grouped as one group), and the other two computational layers are grouped as one group (convolutional layer 2 and the fully connected layer are grouped as one group). Of course, the two adjacent computational layers at the starting end of the neural network can also be grouped as one group (convolutional layer 1 and convolutional layer 2 are grouped as one group), and the other convolutional layer is grouped as one group (convolutional layer 3 is grouped as one group).

[0085] After the computing layers of the neural network are grouped, the standard output corresponding to each group of computing layers can be pre-calculated by the digital computing module 30. Specifically, the standard output is calculated based on the grouping of a number of sequentially connected computing layers, using the digital computing module 30, the trained neural network and the calibration data set to pre-calculate the ideal output result corresponding to each group of computing layers in the trained neural network.

[0086] Here, it should be noted that the calibration data set used to calculate the standard output is the same calibration data set as the calibration data set used to train the neural network, the trained neural network has the same network structure as the neural network during training, and the output results of the trained neural network are consistent with the predicted results.

[0087] After the standard output is calculated, the calculated standard output may be stored in a random access memory (RAM), or the calculated standard output may be stored in a read-only memory (ROM).

[0088] Thereafter, the calibration data set is input from the starting end position of the neural network, that is, the in-memory computing chip 20 receives the calibration data set. After receiving the calibration data set, the in-memory computing chip 20 further calculates the actual output of the group of the computing layer receiving the calibration data set according to the calibration data set and the weight value of the computing layer of the received calibration data set. After the actual output is calculated, the digital computing module 30 calculates the difference between the actual output of the group of the computing layer receiving the calibration data set and the corresponding standard output according to the loss function. When the difference meets the requirement, the in-memory computing chip 20 freezes the weight value corresponding to the computing layer receiving the calibration data set whose difference meets the requirement, and inputs the actual output of the group of the computing layer receiving the calibration data set whose difference meets the requirement as a new calibration data set to the next connected computing layer, and executes the next connected computing layer from the above S140. When the difference does not meet the requirement, the digital computing module 30 uses the gradient descent algorithm to calculate the adjustment increment of the weight value of the computing layer receiving the calibration data set whose difference does not meet the requirement, and adjusts the weight value of the computing layer receiving the calibration data set whose difference does not meet the requirement according to the adjustment increment of the weight value.

[0089] After the weight value is adjusted, the above S140 is executed until the gap meets the requirement.

[0090] Here, it should be noted that freezing the weight value corresponding to the calculation layer that meets the gap requirement and receives the calibration data set means deploying the weight value corresponding to the calculation layer that meets the gap requirement and receives the calibration data set as the trained weight value into the neural network.

[0091] In this embodiment, after the standard output is calculated, a loss function may be constructed in the digital calculation module 30 according to the standard output, and the loss function may include a mean square error (MSE).

[0092] Example 1: Figure 8 When the neural network shown in the figure is trained, the standard outputs of convolution layer 1, convolution layer 2 and the fully connected layer are first calculated based on the calibration data set, and then the calibration data set is input into the in-memory computing chip 20, and the weight value of convolution layer 1 is first adjusted. Fig.10 As shown, the in-memory computing chip 20 receives the calibration data set and calculates the actual output of the convolution layer 1 according to the weight value of the convolution layer 1. The digital computing module 30 calculates the difference between the actual output of the convolution layer 1 and the standard output corresponding to the convolution layer 1. When the difference does not meet the requirements, the digital computing module 30 calculates the adjustment increment of the weight value of the convolution layer 1, and adjusts the weight value of the convolution layer 1 according to the adjustment increment. After the weight value is adjusted once, the in-memory computing chip 20 calculates the actual output of the convolution layer 1 again according to the calibration data set and the adjusted weight value. The digital computing module 30 calculates the difference between the actual output of the convolution layer 1 and the standard output corresponding to the convolution layer 1 again. When the difference does not meet the requirements, the above process is repeated until the difference between the actual output of the convolution layer 1 and the corresponding standard output meets the requirements. When the difference meets the requirements, the in-memory computing chip 20 freezes the weight value of the convolution layer 1. At this point, the weight value adjustment of the convolution layer 1 is completed.

[0093] like Fig.11 As shown, after the weight value of convolution layer 1 is adjusted, the actual output of convolution layer 1 after the weight value adjustment is used as the calibration data set of convolution layer 2 to adjust the weight value of convolution layer 2. The adjustment process of the weight value of convolution layer 2 can refer to the adjustment process of the weight value of convolution layer 1 mentioned above, which will not be repeated here.

[0094] like Fig.12 As shown, after the weight value of convolution layer 2 is adjusted, the actual output of convolution layer 2 with adjusted weight value is used as the calibration data set of the fully connected layer to adjust the weight value of the fully connected layer. The adjustment process of the weight value of the fully connected layer can refer to the adjustment process of the weight value of the convolution layer 1 mentioned above, and will not be repeated here.

[0095] After the weight value of the fully connected layer is adjusted, the weight value of the neural network is adjusted, that is, the training of the neural network is completed.

[0096] Here, it should be noted that the standard outputs of convolution layer 1, convolution layer 2 and the fully connected layer can be pre-calculated based on the grouping of convolution layer 1, convolution layer 2 and the fully connected layer using the digital computing module 30, the trained neural network and the calibration data set, and the calculated standard outputs are stored in a random access memory or a read-only memory.

[0097] Example 2: Fig.13 When the neural network shown is trained, Fig.13 In the example, convolution layer 1 is a group, and convolution layer 2 and the fully connected layer are a group. When calculating the standard output, the standard output corresponding to the convolution layer 1 is calculated by the digital computing module 30 according to the calibration data set in units of groups. The digital computing module 30 calculates the standard output of the convolution layer 2 and the fully connected layer according to the calibration data set (compared with Example 1, only one standard output needs to be calculated here, that is, one standard output is used when the weight value of the convolution layer 2 and the fully connected layer is adjusted). When training is performed specifically, the digital computing module 30 first calculates the standard output of the convolution layer 1, the convolution layer 2 and the fully connected layer according to the calibration data set (the convolution layer 1 corresponds to one standard output, and the convolution layer 2 and the fully connected layer correspond to one standard output). Thereafter, the calibration data set is input into the in-memory computing chip 20 to adjust the weight value of the convolution layer 1. The specific process refers to the weight value adjustment process in Example 1, which will not be repeated here.

[0098] After the weight value of convolution layer 1 is adjusted, when the weight values ​​of convolution layer 2 and the fully connected layer are adjusted, the actual output of convolution layer 1 after the weight value adjustment is used as the calibration data set for convolution layer 2 and the fully connected layer to adjust the weight values ​​of convolution layer 2 and the fully connected layer. The specific adjustment process refers to the weight value adjustment process in Example 1. The difference is that when the gap between the actual output of the fully connected layer and the standard output corresponding to convolution layer 2 and the fully connected layer does not meet the requirements, the weight value of convolution layer 2 and the weight value of the fully connected layer are adjusted at the same time.

[0099] Compared with Example 1, Example 2 requires fewer standard outputs. Example 1 requires three standard outputs, while Example 2 requires two standard outputs.

[0100] In this embodiment, the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning adjusts the weight value in a grouped manner. When the gap between the actual output and the standard output of a group of computing layers does not meet the requirements, only the weight value corresponding to the gap between the actual output and the standard output of the group of computing layers that does not meet the requirements is adjusted. Only a small amount of calibration data set is required (there is no need to adjust the weight values ​​of all computing layers every time the weight value is adjusted), which reduces the use of memory and the demand for calibration data sets. In addition, there is no need to add a calibrator module, which reduces the complexity and power consumption of the hardware. At the same time, the calculation delay caused by the calibrator module and the need for a large-scale calibration data set to avoid overfitting are avoided. Because the calibration data set is input from the computing layer at the starting position in the neural network, the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning can complete the training of the neural network.

[0101] In addition, in the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning, the result calculated by the digital computing module 30 is used as the standard result, so that the actual output of the neural network is closer to the ideal result.

[0102] In this embodiment, the calibration data set may be stored in a random access memory (RAM), and the calculated standard output may be stored in a read-only memory (ROM).

[0103] Furthermore, the digital computing module 30 can be placed on an internal computing chip. In this way, when the weight value stored on the internal computing chip changes, the weight value can be adjusted automatically on the internal computing chip without the need for other external hardware resources.

[0104] In some embodiments, the digital computing module 30 of the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning is also used to pre-train the weight values ​​of several computing layers and deploy the weight values ​​on the in-memory computing chip 20, that is, the weight values ​​of several computing layers can be pre-trained by the digital computing module as the initial weight values ​​when training the neural network.

[0105] The digital computing module 30 may be, but is not limited to, an image processor. Before training the neural network, the image processor may pre-calculate the weight values ​​corresponding to each computing layer based on the specific network structure of the neural network, and deploy the weight values ​​to the in-memory computing chip 20.

[0106] In order to solve some problems existing in the process of adjusting the weight value of the in-memory computing chip 20, such as slow writing speed of adjusting the weight value and limited erase and write life, further, as Figure 6 , Fig.14As shown, the on-chip fine-tuning device on the in-memory computing chip 20 based on group-by-group fine-tuning also includes a digital fine-tuning auxiliary module 40, which is used to obtain a single adjustment increment when adjusting the weight value; the digital fine-tuning auxiliary module 40 is also used to accumulate the adjustment increments when adjusting the weight value obtained n times to the weight value of the calculation layer of the calibration data set received whose gap does not meet the requirements after obtaining n adjustment increments to adjust the weight value, wherein n is a positive integer; wherein, after the weight value is adjusted, execution starts from the above S140 until the gap meets the requirements.

[0107] Specifically, when adjusting the weight value, after calculating the adjustment increment of the weight value, the digital fine-tuning auxiliary module 40 is used to obtain a single adjustment increment of the weight value, and after obtaining n adjustment increments, the adjustment increments of the weight values ​​obtained n times are added to the weight value of the calculation layer whose gap does not meet the requirements of receiving the calibration data set, so as to adjust the weight value, wherein n can be 1, 2, 3, 4...8, 9, 10...etc., and technicians in this field can determine the specific value of n according to actual needs, and no excessive restrictions are made here.

[0108] Exemplarily, when adjusting the weight value, if the calculated single adjustment increment of the weight value is 1, after obtaining n adjustment increments, the adjustment increments of the weight value obtained n times are accumulated to the weight value of the calculation layer whose gap does not meet the requirements for receiving the calibration data set, that is, n 1s are added and accumulated to the weight value of the calculation layer whose gap does not meet the requirements for receiving the calibration data set.

[0109] In this embodiment, the weight value can be adjusted faster by adjusting the weight value through the digital fine-tuning auxiliary module 40, and the in-memory computing chip 20 has a long service life (the non-volatile memory in the in-memory computing chip 20 has a limited erasable life).

[0110] The digital fine-tuning auxiliary module 40 may be a static random access memory.

[0111] In some embodiments, Fig.15 As shown, after the neural network training is completed, the neural network is further inferred, wherein, during the inference process of the neural network, there is no need for a digital fine-tuning auxiliary module 40, so that no additional hardware is required during the inference process, which reduces hardware costs and power consumption and achieves more efficient operations.

[0112] In some embodiments, Figure 4As shown, the present invention also provides an on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning, and the on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning can be applied to the on-chip fine-tuning device on the in-memory computing chip based on group-by-group fine-tuning; wherein, the on-chip fine-tuning method is used to train a neural network, and the neural network includes a plurality of computing layers connected in sequence, and the plurality of computing layers include a convolutional layer and a fully connected layer which is the output end of the neural network.

[0113] The on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning may include the steps of:

[0114] S110, grouping a plurality of computing layers in the neural network, wherein the plurality of computing layers are divided into at least two groups;

[0115] S120, based on the grouping of a plurality of sequentially connected computing layers and the calibration data set, calculating the standard output corresponding to each group of computing layers;

[0116] S130, receiving a calibration data set;

[0117] S140, calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set;

[0118] S150, calculating the difference between the actual output of the group of the computing layer that receives the calibration data set and the corresponding standard output according to the loss function;

[0119] S160, when the gap meets the requirement, freeze the weight value corresponding to the computing layer that meets the gap requirement and receives the calibration data set, and use the actual output of the group of computing layers that meet the gap requirement and receive the calibration data set as a new set of calibration data sets to input to the next connected computing layer, and execute the next connected computing layer from S140 above;

[0120] S170, when the gap does not meet the requirement, using a gradient descent algorithm to calculate an adjustment increment of a weight value of a calculation layer of the calibration data set received for which the gap does not meet the requirement, and adjusting the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirement according to the adjustment increment of the weight value;

[0121] After the weight value is adjusted, the above S140 is executed until the gap meets the requirement;

[0122] Among them, the calibration dataset is input from the computing layer at the starting position in the neural network.

[0123] In this embodiment, when the neural network is layered, those skilled in the art may group according to actual needs when the number of grouping layers is at least two. For example, one computing layer may be grouped into one group, two sequentially connected computing layers may be grouped into one group, or three sequentially connected computing layers may be grouped into one group.

[0124] After the computing layers of the neural network are grouped, the standard output corresponding to each group of computing layers can be pre-calculated by the digital computing module. Specifically, the standard output is calculated based on the grouping of several sequentially connected computing layers, using the digital computing module, the trained neural network and the calibration data set to pre-calculate the ideal output result corresponding to each group of computing layers in the trained neural network.

[0125] Here, it should be noted that the calibration data set used to calculate the standard output is the same calibration data set as the calibration data set used to train the neural network, the trained neural network has the same network structure as the neural network during training, and the output results of the trained neural network are consistent with the predicted results.

[0126] After the standard output is calculated, the calculated standard output may be stored in a random access memory (RAM), or the calculated standard output may be stored in a read-only memory (ROM).

[0127] Thereafter, the calibration data set is input from the starting end position of the neural network, that is, the in-memory computing chip receives the calibration data set. After receiving the calibration data set, the in-memory computing chip further calculates the actual output of the group of the computing layer receiving the calibration data set according to the calibration data set and the weight value of the computing layer of the received calibration data set. After the actual output is calculated, the digital computing module calculates the difference between the actual output of the group of the computing layer receiving the calibration data set and the corresponding standard output according to the loss function. When the difference meets the requirement, the in-memory computing chip freezes the weight value corresponding to the computing layer receiving the calibration data set whose difference meets the requirement, and inputs the actual output of the group of the computing layer receiving the calibration data set whose difference meets the requirement as a new calibration data set to the next connected computing layer, and executes the next connected computing layer from the above S140. When the difference does not meet the requirement, the digital computing module uses the gradient descent algorithm to calculate the adjustment increment of the weight value of the computing layer receiving the calibration data set whose difference does not meet the requirement, and adjusts the weight value of the computing layer receiving the calibration data set whose difference does not meet the requirement according to the adjustment increment of the weight value.

[0128] After the weight value is adjusted, the above S140 is executed until the gap meets the requirement.

[0129] Here, it should be noted that freezing the weight value corresponding to the calculation layer that meets the gap requirement and receives the calibration data set means deploying the weight value corresponding to the calculation layer that meets the gap requirement and receives the calibration data set as the trained weight value into the neural network.

[0130] In this embodiment, after the standard output is calculated, a loss function may be constructed in the digital calculation module according to the standard output, and the loss function may include a mean square error (MSE).

[0131] In this embodiment, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning adjusts the weight values ​​in a grouped manner. When the gap between the actual output and the standard output of a group of computing layers does not meet the requirements, only the weight values ​​corresponding to the gap between the actual output and the standard output of the group of computing layers that does not meet the requirements are adjusted. Only a small amount of calibration data set is required, which reduces the use of memory and the demand for calibration data sets. In addition, there is no need to add a calibrator module, which reduces the complexity and power consumption of the hardware. At the same time, the calculation delay caused by the calibrator module and the need for a large-scale calibration data set to avoid overfitting are avoided. Because the calibration data set is input from the computing layer at the starting position in the neural network, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning can complete the training of the neural network.

[0132] In some embodiments, Figure 4 As shown, before calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set, the steps include:

[0133] S100, pre-training weight values ​​of several computing layers, and deploying the weight values ​​on an in-memory computing chip.

[0134] In this step, the weight values ​​of several computing layers are pre-trained by a digital computing module, and the weight values ​​are deployed on the in-memory computing chip. The digital computing module may be, but is not limited to, an image processor. The details are as described in the above-mentioned embodiment of the on-chip fine-tuning device on the in-memory computing chip based on group-by-group fine-tuning, and will not be described in detail here.

[0135] In some embodiments, Figure 5 As shown, when the gap does not meet the requirements, the gradient descent algorithm is used to calculate the adjustment increment of the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirements, and the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirements is adjusted according to the adjustment increment of the weight value, and after the weight value is adjusted, the step of starting from the above S140 until the gap meets the requirements includes the steps of:

[0136] S171, obtaining a single adjustment increment when adjusting the weight value;

[0137] S172. After obtaining n adjustment increments, the adjustment increments obtained n times when adjusting the weight value are accumulated to the weight value of the calculation layer of the calibration data set received whose gap does not meet the requirement to adjust the weight value, where n is a positive integer.

[0138] In this embodiment, when adjusting the weight value, after the digital calculation module calculates the adjustment increment of the weight value, the single adjustment increment of the weight value is obtained through the digital auxiliary fine-tuning module, and after obtaining n adjustment increments, the adjustment increment of the weight value obtained n times is accumulated to the weight value of the calculation layer of the calibration data set received whose gap does not meet the requirements, so as to adjust the weight value, wherein n can be 2, 3, 4...8, 9, 10...etc., and those skilled in the art can determine the specific value of n according to actual needs, and no excessive limitation is made here. Specifically, as described in the above-mentioned embodiment of the on-chip fine-tuning device on the in-memory computing chip based on group-by-group fine-tuning, no excessive elaboration is given here.

[0139] In some embodiments, the loss function may be a mean square error. Specifically, after the standard output is calculated, a loss function may be constructed in the digital computing module according to the standard output, and the loss function may include a mean square error (MSE). As described in the above embodiment of the on-chip fine-tuning device on the in-memory computing chip based on group-by-group fine-tuning, it will not be described in detail here.

[0140] In some embodiments, Figure 4 As shown, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning also includes the steps of:

[0141] S180: After the weight values ​​of the computing layers of each group in the neural network are adjusted, the neural network is inferred using the adjusted weight values.

[0142] In this step, in the reasoning process of the neural network, no digital auxiliary fine-tuning module is required, so that no additional hardware is required in the reasoning process, which reduces hardware cost and power consumption and achieves more efficient operation. As described in the above embodiment of the on-chip fine-tuning device on the in-memory computing chip based on group-by-group fine-tuning, it will not be described in detail here.

[0143] In summary, the present invention provides an on-chip fine-tuning method and device on an in-memory computing chip based on group-by-group fine-tuning, which has the following beneficial effects:

[0144] When adjusting the weight value, the adjustment is performed in groups. When the gap between the actual output and the standard output of a group of computing layers does not meet the requirements, only the weight value corresponding to the gap between the actual output and the standard output of the group of computing layers that does not meet the requirements is adjusted. Only a small amount of calibration data set is required, which reduces the use of memory and the demand for calibration data sets. In addition, there is no need to add a calibrator module, which reduces the complexity and power consumption of the hardware. At the same time, the calculation delay caused by the calibrator module is avoided, and there is no need for a large-scale calibration data set to avoid overfitting. Because the calibration data set is input from the computing layer at the starting position in the neural network, the on-chip fine-tuning method on the in-memory computing chip based on group-by-group fine-tuning can complete the training of the neural network.

[0145] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the scope covered by the claims of the present invention.

Claims

1. An on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning, for training a neural network, the neural network comprising a plurality of sequentially connected computing layers, the plurality of computing layers comprising a convolutional layer and a fully connected layer as an output end of the neural network, characterized in that: The on-chip fine-tuning method comprises: Step A, grouping a plurality of computing layers in the neural network, wherein the plurality of computing layers are divided into at least two groups; Step B, based on the grouping of the plurality of sequentially connected computing layers and the calibration data set, calculating the standard output corresponding to each group of computing layers; Step C, receiving a calibration data set; Step D, calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set; Step E, calculating the difference between the actual output of the group of computing layers that received the calibration data set and the corresponding standard output according to the loss function; When the gap meets the requirement, the weight value corresponding to the computing layer that meets the requirement and receives the calibration data set is frozen, and the actual output of the group of computing layers that meet the requirement and receive the calibration data set is input as a new set of calibration data sets to the next connected computing layer, and the next connected computing layer is executed from step D; When the gap does not meet the requirements, a gradient descent algorithm is used to calculate the adjustment increment of the weight value of the calculation layer that receives the calibration data set, and the weight value of the calculation layer that receives the calibration data set is adjusted according to the adjustment increment of the weight value; After the weight value is adjusted, the process starts from step D until the gap meets the requirement; The calibration data set is input from the computing layer at the starting position in the neural network.

2. The on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning according to claim 1, characterized in that: The step of calculating the actual output of the group to which the calculation layer receiving the calibration data set belongs according to the calibration data set and the weight value of the calculation layer receiving the calibration data set comprises: Pre-train the weight values ​​of several computing layers, and deploy the weight values ​​on the in-memory computing chip.

3. The on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning according to claim 1, characterized in that: When the gap does not meet the requirement, the gradient descent algorithm is used to calculate the adjustment increment of the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirement, the weight value of the calculation layer of the calibration data set received for which the gap does not meet the requirement is adjusted according to the adjustment increment of the weight value, and the step D is started after the weight value is adjusted until the gap meets the requirement, including: Get the single adjustment increment when adjusting the weight value; After obtaining n adjustment increments, the adjustment increments of the adjusted weight values ​​obtained n times are added to the weight value of the calculation layer of the calibration data set received when the gap does not meet the requirements to adjust the weight value, and after the weight value is adjusted, execution is started from the step D until the gap meets the requirements, where n is a positive integer.

4. The on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning according to claim 1, characterized in that: The standard output is an ideal output result corresponding to each group of the computing layers in the neural network, which is pre-calculated based on the grouping of several sequentially connected computing layers using a digital computing module, a trained neural network and a calibration data set.

5. The on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning according to claim 1, characterized in that: The loss function includes the mean square error.

6. The on-chip fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning according to claim 1, characterized in that: After the weight values ​​of the computing layers of each group in the neural network are adjusted, the neural network is inferred using the adjusted weight values.

7. An on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning, for training a neural network, the neural network comprising a plurality of sequentially connected computing layers, the plurality of computing layers comprising a convolutional layer and a fully connected layer as an output end of the neural network, characterized in that: The on-chip fine-tuning device comprises: A neural network grouping module, used for grouping a plurality of computing layers in the neural network on the in-memory computing chip, wherein the plurality of computing layers are divided into at least two groups; A digital calculation module, used for calculating the standard output corresponding to each group of the calculation layers based on the grouping conditions of the plurality of sequentially connected calculation layers and the calibration data set; An in-memory computing chip for receiving a calibration data set; The in-memory computing chip is also used to perform step D in the fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning as claimed in any one of claims 1 to 6; The digital computing module is also used to perform step E in the fine-tuning method on an in-memory computing chip based on group-by-group fine-tuning as claimed in any one of claims 1 to 6; The in-memory computing chip is also used to freeze the weight value corresponding to the computing layer that receives the calibration data set when the gap meets the requirement, and input the actual output of the group of computing layers that receive the calibration data set as a new calibration data set to the next connected computing layer, and execute the next connected computing layer from step D. The digital calculation module is also used to use a gradient descent algorithm to calculate the adjustment increment of the weight value of the calculation layer of the calibration data set received by the gap not meeting the requirement when the gap does not meet the requirement, and adjust the weight value of the calculation layer of the calibration data set received by the gap not meeting the requirement according to the adjustment increment of the weight value; After the weight value is adjusted, the process starts from step D until the gap meets the requirement; The calibration data set is input from the computing layer at the starting position in the neural network.

8. The on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning according to claim 7, characterized in that: The on-chip fine-tuning device further comprises: The digital computing module is used to pre-train the weight values ​​of several computing layers and deploy the weight values ​​on the in-memory computing chip.

9. The on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning according to claim 7, characterized in that: The on-chip fine-tuning device further comprises: A digital assisted fine-tuning module is used to obtain a single adjustment increment when adjusting the weight value; It is also used to accumulate the adjustment increments when adjusting the weight values ​​obtained n times to the weight value of the calculation layer of the calibration data set received when the gap does not meet the requirements after obtaining n adjustment increments to adjust the weight value, and start executing from the step D after the weight value is adjusted until the gap meets the requirements, where n is a positive integer.

10. The on-chip fine-tuning device on an in-memory computing chip based on group-by-group fine-tuning according to claim 8, characterized in that: The digital computing module is a graphics processor.

Images