Neural network layer training method, neural network computing system and computer readable medium

Abstract

The present invention provides a layer training method for a neural network, wherein the layer includes one or more weight parameters stored with a first precision, and the training method includes: S101: providing input to the layer; S102: selecting a second precision lower than the first precision; S103: quantizing the one or more weight parameters with the second precision; S104: calculating the output of the layer based on the input with the one or more weight parameters of the second precision; S105: updating the one or more weight parameters with the first precision through a back propagation algorithm. The present invention proposes the concept and implementation of an arbitrary precision neural network. This allows the same model to be dynamically quantized to different precision levels during deployment. There is no need to maintain the corresponding model for the required precision level, and in actual use, more abundant precision level options are provided to avoid the problem of not being able to meet the requirements.

Description

Technical Field

[0001] The present invention generally relates to the field of artificial intelligence, and more particularly to a method for training a neural network layer, a neural network computing system with adjustable precision, and a computer-readable medium. Background Art

[0002] Neural networks have certain requirements for computing power. Given a trained neural network, if the computing speed needs to be guaranteed under limited computing power, the accuracy needs to be sacrificed. In practical applications, the computing power may be dynamically adjustable. For example, a mobile phone can provide high computing power when the battery level is high, and needs to reduce the computing power to save energy when the battery level is low. In this case, a model that can dynamically balance computing power and accuracy is needed to meet the actual application requirements.

[0003] Existing technical solutions need to maintain multiple different models, prepare models at each precision level for different running precisions, and replace the models according to needs during deployment to meet application requirements. As Figures 1A-1C shown, among which Figure 1A shows a neural network model with low precision, where the computing precision is the lowest, for example, 1 bit; Figure 1B shows a neural network model with high precision, where the computing precision is the highest, for example, 32 bit; Figure 1C shows a neural network model with intermediate precision, where the computing precision is in the middle, for example, n1 bit, where 1 < n1 < 32. According to needs, more neural network models with intermediate precision can also be prepared and stored in advance. During calculation, Figure 1A the neural network model with low precision requires the lowest power or computing power, and has the worst accuracy at the same time; Figure 1B the neural network model with high precision requires the highest power or computing power, and has the highest accuracy at the same time; Figure 1C the neural network model with intermediate precision requires the intermediate power or computing power, and has the intermediate accuracy at the same time.

[0004] The main problems of the existing technology are high storage space cost and lack of flexibility in deployment. In the model preparation stage, the different precision levels required in the application must be determined, and models are trained for each precision level. During deployment, different models need to be loaded according to the precision requirements. If a precision level that has not been prepared in advance is encountered, a model that can meet the needs cannot be provided. In the case of a large number of precision levels, multiple models need to be stored, resulting in a high demand for storage space. Summary of the Invention

[0005] In view of at least one problem of the existing technology, the present invention provides a method for training a layer of a neural network, where the layer includes one or more weight parameters stored at a first precision, and the training method includes:

[0006] S101: providing input to the layer;

[0007] S102: selecting a second precision lower than the first precision;

[0008] S103: quantizing the one or more weight parameters with the second precision;

[0009] S104: Calculate the output of the layer based on the input using the one or more weight parameters of the second precision;

[0010] S105: Update the one or more weight parameters with the first precision through a back-propagation algorithm.

[0011] According to one aspect of the present invention, the layer training method further includes: adjusting the second precision, and repeating steps S101, S102, S103 and S104.

[0012] According to one aspect of the present invention, step S102 includes: randomly selecting a second precision lower than the first precision.

[0013] According to one aspect of the present invention, the layer training method further includes: calculating the output of the layer based on the input using the one or more weight parameters of the first precision.

[0014] According to one aspect of the present invention, the layer training method further comprises: training all layers of the neural network through steps S101, S102, S103, S104 and S105.

[0015] The present invention also provides a neural network computing system with adjustable precision, comprising an IO interface, a memory, a quantizer and a neural network chip, wherein

[0016] The IO interface is configured to input / output data;

[0017] The memory is configured to store a neural network model and weight parameters of the neural network model, wherein the weight parameters are stored with a first precision;

[0018] The quantizer is in communication with the memory and can quantize and output the weight parameter into a second precision or a first precision, wherein the second precision is lower than the first precision;

[0019] The neural network chip is configured to perform artificial neural network operations according to the weight parameters of the first precision or the second precision.

[0020] According to one aspect of the present invention, the quantizer is configured to select a first precision or a different second precision to quantize the weight parameters according to the computing power and / or power of the neural network chip.

[0021] According to one aspect of the present invention, the quantizer is configured to select a lower first precision to quantize the weight parameters as the computing power and / or power of the neural network chip decreases.

[0022] According to one aspect of the present invention, the neural network chip is configured to execute a back-propagation training algorithm and update the weight parameters with a first accuracy.

[0023] The present invention also provides a computer-readable medium having instructions stored thereon, wherein when the instructions are executed by a processor, the layer training method as described above can be implemented.

[0024] The present invention proposes the concept and implementation of arbitrary precision neural network. It enables the same model to be dynamically quantized to different precision levels during deployment. There is no need to maintain the corresponding model for the required precision level. At the same time, in actual use, more abundant precision boundary options are provided to avoid the problem of not being able to meet the needs. It has been verified that the accuracy of the network trained by the proposed method at different precisions is maintained at the same level as the accuracy of the model trained at this precision level alone. Training the neural network at different precisions enables the obtained parameters to be dynamically adjusted to different precisions for deployment. By maintaining batch normalization parameters at different precisions, it is obtained that this training method can converge and obtain the required arbitrary precision neural network. By dynamically modifying the structure of the network model, a similar effect of dynamically balancing the computing power demand and the accuracy demand can be achieved. However, it is a method that does not conflict with this method, that is, the two methods can be used at the same time. Compared with the previous method, the accuracy of each layer of the neural network is fixed after training. The proposed arbitrary precision neural network supports dynamic adjustment of precision. In order to realize the arbitrary precision neural network, the present invention proposes a method for jointly training parameters of different precisions during training. For each parameter in the model, a full-precision floating-point value is maintained. During training, in each forward propagation process, the parameters are pre-quantized to a randomly selected precision level, and the gradient is updated to this full-precision floating-point value during backpropagation. This allows the parameters obtained during the network training process to be dynamically adjusted during deployment. Considering that most of the current commonly used neural network structures contain batch normalization (BatchNorm) operations. The scheme needs to maintain different batch normalization parameters for different precisions. The number of this parameter is very small compared to the total number of neural networks, so it will not cause a significant change in the storage size. It has been verified that the accuracy of the network trained using the proposed method at different precisions is at the same level as the accuracy of the model trained at this precision level alone. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0026] Figure 1A A low-precision neural network model is shown;

[0027] Figure 1B A high-precision neural network model is shown;

[0028] Figure 1C A neural network model of intermediate accuracy is shown;

[0029] Figure 2 A layer training method of a neural network according to an embodiment of the present invention is shown;

[0030] Figure 3 A graphical illustration of training the layers is shown;

[0031] Figure 4 A schematic diagram showing quantization of weights according to a second precision; and

[0032] Figure 5 A neural network computing system with adjustable precision according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0034] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or can communicate with each other; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply means that the first feature is lower in level than the second feature.

[0037] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplicity and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides various specific examples of processes and materials, but those of ordinary skill in the art can be aware of the application of other processes and / or the use of other materials.

[0038] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0039] A typical neural network includes multiple layers. The input layer is used to receive input, the output layer is used to output calculation results, and the intermediate layer (or hidden layer) is between the input layer and the output layer. Each layer can include one or more units (or neurons), and the units in adjacent layers are connected. Each connection has a corresponding weight (its value is called weight), and the weight needs to be obtained and optimized through neural network training. The training of a neural network is to adjust the value of the weight to the best so that the prediction effect of the entire network is the best. In the training of existing neural networks, forward calculations and back propagation are performed with fixed precision weights. The inventor of the present invention provides a variable precision training method and a neural network model obtained thereby. Given a model structure of a neural network, the training process of an arbitrary precision neural network is as follows:

[0040] For a certain layer in the neural network, during forward calculation, the network parameters of the maintained floating-point values ​​are first quantized to n-bit precision (which is lower than the precision of the maintained floating-point values), where n is a randomly selected value.

[0041] Then use the low-precision weights and inputs to calculate the corresponding input according to the calculation method of this specific layer, which serves as the input of the next layer;

[0042] When backpropagation updates the network parameters, the returned gradients are updated to the floating-point value version of the weights.

[0043] Because of the randomly selected low precision introduced during the training process, the network can learn to adapt to this randomness during training, so that arbitrary precision can be supported when the trained model is used for calculations.

[0044] Figure 2 A neural network layer training method 100 according to an embodiment of the present invention is shown. Figure 3 A diagram illustrating training the layer is shown. The layer includes one or more weight parameters stored in a first precision, and the layer may be any layer in a multi-layer neural network. The following is a detailed description with reference to the accompanying drawings. Figure 3 As shown, the first precision is, for example, a 32-bit floating point value, that is, all weights in this layer are stored as 32-bit floating point values.

[0045] like Figure 2 The training method 100 comprises:

[0046] In step S101: providing input to the layer. The input may include various types of input such as pictures, audio, video, numerical vectors, etc., all of which are within the protection scope of the present invention.

[0047] In step S102: a second precision lower than the first precision is selected.

[0048] Taking the first precision as a 32-bit floating point value as an example, the second precision can be randomly selected as long as it is lower than the first precision. Preferably, the second precision is randomly selected from 1, 2, 4, 8, 16-bit floating point values.

[0049] In step S103: quantizing the one or more weight parameters with the second precision to obtain low-precision numerical network parameters.

[0050] like Figure 3 As shown in the figure, after selecting the second precision, the weight parameters in the layer are quantized with the second precision. For a parameter in the network, the parameter under different precisions can be obtained simply by performing a shift operation. Figure 4 Detailed description. Assuming that the first precision is an 8-bit floating point number, the binary representation of one weight is 10010100. The available second precisions include 1-, 2-, and 4-bit floating point numbers. When a 4-bit floating point number is selected as the second precision, the weight is quantized to 1001 (binary); when a 2-bit floating point number is selected as the second precision, the weight is quantized to 10 (binary); when a 1-bit floating point number is selected as the second precision, the weight is quantized to 1 (binary).

[0051] In step S104: the output of the layer is calculated based on the input with the one or more weight parameters of the second precision.

[0052] like Figure 3 As shown, after quantizing the one or more weight parameters with the second precision to obtain low-precision numerical network parameters, based on the input in step S101, the forward operation (forward propagation) of this layer of the neural network is performed to obtain the output of the forward calculation.

[0053] In step S105: the one or more weight parameters are updated with the first precision through a back propagation algorithm.

[0054] The purpose of neural network model training is to make the weights corresponding to each connection as close to the real model as possible. The specific method is as follows. The predicted target of the sample is yp, and the real target is y. Then define a value loss, and the calculation formula is as follows.

[0055] loss = (yp-y) 2

[0056] This value loss is called loss, and the goal of training is to make the sum of losses for all training data as small as possible. Generally speaking, the gradient descent algorithm is used to solve this optimization problem. The gradient descent algorithm calculates the current gradient of the parameter each time, and then allows the parameter to move a distance in the opposite direction of the gradient, and repeats it continuously until the gradient is close to zero. Generally, at this time, all parameters just reach a state that makes the loss function reach a minimum value. In the neural network model, due to the complex structure, the cost of calculating the gradient each time is very high. Therefore, the back propagation algorithm is also needed. The back propagation algorithm uses the structure of the neural network for calculation. Instead of calculating the gradients of all parameters at once, it is from back to front. First, the gradient of the output layer is calculated, then the gradient of the second parameter matrix, followed by the gradient of the middle layer, then the gradient of the first parameter matrix, and finally the gradient of the input layer. After the calculation is completed, the gradients of the two required parameter matrices are available.

[0057] like Figure 3 As shown, the one or more weight parameters are updated with a first accuracy through a back-propagation algorithm.

[0058] The above process completes a forward calculation and back propagation for the neural network model. In addition, the second precision can be adjusted, and the steps S101, S102, S103, S104, and S105 can be repeated. For example, in the above process, an 8-bit floating point number is selected as the second precision, then in the next calculation process, a 4-bit floating point number is selected as the second precision to recalculate.

[0059] In addition, in addition to performing the above calculations with the second precision, the layer training method 100 further includes: calculating the output of the layer based on the input with the one or more weight parameters of the first precision. Therefore, in the above training process, the neural network model is trained with high precision and one or more low precision weights, respectively.

[0060] The above describes an embodiment of training one layer of a neural network. Those skilled in the art will readily appreciate that the above layer training method can also be applied to multiple layers or all layers of a neural network, and all layers of the neural network that need to be trained can be trained through steps S101, S102, S103, S104, and S105.

[0061] The above layer training method helps to obtain multi-precision or arbitrary precision neural networks. By jointly training parameters of different precisions during the neural network training process, the obtained neural network model can adapt to network parameters or weights of different precisions. For each parameter in the model, a full-precision floating-point value is maintained. During training, in each forward propagation process, the parameters are pre-quantized to a randomly selected precision level, and the gradient is updated to this full-precision floating-point value during backpropagation. This allows the parameters obtained during the network training process to be dynamically adjusted when deployed.

[0062] like Figure 5 As shown, the present invention also relates to a neural network computing system 200 with adjustable precision, including an IO interface 201, a memory 202, a quantizer 203 and a neural network chip 204. The IO interface 201 is configured to input / output data, and the input data provided to the neural network chip 204, as well as the data calculated and output by the neural network chip 204, can be input and output through the IO interface 201. The memory 202 is configured to store a neural network model and weight parameters of the neural network model, wherein the weight parameters are stored with a first precision. The memory may include a non-volatile memory and a volatile memory, the former can permanently store the neural network model and the weight parameters of the neural network model, and the latter can be used for temporary storage. The quantizer 203 communicates with the memory 202, and can quantize and output the weight parameters to a second precision or a first precision, wherein the second precision is lower than the first precision. For a parameter in the network, the quantizer 203 can simply obtain the parameter at different precisions through a displacement operation, as described in the reference above. Figure 4 The neural network chip 204 is configured to perform artificial neural network operations according to the weight parameters of the first precision or the second precision.

[0063] According to a preferred embodiment of the present invention, the quantizer 203 is configured to select a first precision or a different second precision to quantize the weight parameters according to the computing power and / or power of the neural network chip. For example, when an external power supply is used, the power is sufficient, and a higher precision can be used to quantize the weight parameters for calculation by the neural network chip. When a built-in battery is used as a power supply, power is more precious, so a lower precision is selected to quantize the weight parameters.

[0064] In addition, it is easy for those skilled in the art to understand that in the context of the present invention, the second precision does not necessarily refer to a certain precision, and it may include one precision or any one of multiple precisions. Taking the first precision of the 32-bit floating point number above as an example, the second precision may be one of 1, 2, 4, 8, and 16-bit floating point numbers, and the second precision may also change in different computing environments, all of which are within the protection scope of the present invention.

[0065] According to a preferred embodiment of the present invention, the quantizer is configured to select a lower second precision to quantize the weight parameters as the computing power and / or power of the neural network chip decreases.

[0066] According to a preferred embodiment of the present invention, the neural network chip is configured to execute a back propagation training algorithm and update the weight parameters with a first precision. When an arbitrary precision neural network is actually applied, as shown in the following figure, according to the actual computing power, the network parameters can be first quantized to a low precision such as 1-bit, 2-bit or any precision level lower than the pre-stored one, and then forward calculation is performed to obtain the output.

[0067] The present invention also relates to a computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, can implement the layer training method as described above.

[0068] The arbitrary precision neural network of the present invention is compared with the previous method, in which the accuracy of each layer of the neural network is fixed after training. The proposed arbitrary precision neural network supports dynamic adjustment of accuracy. Considering that most of the structures of currently commonly used neural networks contain batch normalization (BatchNorm) operations. The solution of the present invention needs to maintain different batch normalization parameters for different precisions. The number of this parameter is very small compared to the total number of neural networks, so it will not cause a significant change in the storage size.

[0069] It has been verified that the accuracy of the network trained using the method proposed in the present invention is at the same level as the accuracy of the model trained at this accuracy level alone under different precisions. Training the neural network at different precisions enables the obtained parameters to be dynamically adjusted to different precisions for deployment. By maintaining batch normalization parameters at different precisions, it is obtained that this training method can converge and obtain the required arbitrary precision neural network. By dynamically modifying the structure of the network model, a similar effect of dynamically balancing computing power requirements and accuracy requirements can also be achieved. However, it is a method that does not conflict with this method, that is, the two methods can be used at the same time. Compared with the previous method, the accuracy of each layer of the neural network is fixed after training. The proposed arbitrary precision neural network supports dynamic adjustment of accuracy. In order to realize an arbitrary precision neural network, the present invention proposes a method for jointly training parameters of different precisions during training. For each parameter in the model, a full-precision floating-point value is maintained. During training, in each forward propagation process, the parameters are pre-quantized to a randomly selected precision level, and the gradient is updated to this full-precision floating-point value during back propagation. In this way, the parameters obtained by the network during training can also be dynamically adjusted when deployed. Considering that most of the commonly used neural network structures currently contain batch normalization (BatchNorm) operations. The scheme needs to maintain different batch normalization parameters for different precisions. The number of these parameters is very small compared to the total number of neural networks, so it will not cause a significant change in the storage size. It has been verified that the accuracy of the network trained using the proposed method at different precisions is at the same level as the accuracy of the model trained at this precision level alone.

[0070] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training a layer of a neural network, the layer comprising one or more weight parameters stored with a first precision, the training method comprising: S101: providing input to the layer; S102: Randomly selecting a second precision that is lower than the first precision; S103: quantizing the one or more weight parameters with the second precision; S104: Calculate the output of the layer based on the input using the one or more weight parameters of the second precision; S105: Update the one or more weight parameters with the first precision through a back-propagation algorithm.

2. The layer training method of claim 1, further comprising: The second precision is adjusted, and steps S101, S102, S103, S104 and S105 are repeated.

3. The layer training method according to claim 1 or 2, further comprising: An output of the layer is calculated based on the input with the one or more weight parameters of the first precision.

4. The layer training method according to claim 1 or 2, further comprising: All layers of the neural network are trained through steps S101, S102, S103, S104 and S105.

5. A neural network computing system with adjustable precision, comprising an IO interface, a memory, a quantizer and a neural network chip, wherein The IO interface is configured to input / output data; The memory is configured to store a neural network model and weight parameters of the neural network model, wherein the weight parameters are stored with a first precision; The quantizer is in communication with the memory and can quantize and output the weight parameter into a second precision or a first precision, wherein the second precision is lower than the first precision and the second precision is randomly selected; The neural network chip is configured to perform artificial neural network operations according to the weight parameters of the first precision or the second precision.

6. A neural network computing system as described in claim 5, wherein the quantizer is configured to select a first precision or a different second precision to quantize the weight parameters according to the computing power and / or power of the neural network chip.

7. A neural network computing system as described in claim 6, wherein the quantizer is configured to: select a lower second precision to quantize the weight parameters as the computing power and / or power of the neural network chip decreases.

8. A neural network computing system as described in any one of claims 5-7, wherein the neural network chip is configured to execute a back-propagation training algorithm and update the weight parameters with a first accuracy.

9. A computer-readable medium having instructions stored thereon, wherein when the instructions are executed by a processor, the layer training method according to any one of claims 1 to 4 can be implemented.

Images