Improved quantification-sensitive learning method for a neural network

Abstract

Improved quantization-sensitive learning method for a neural network Learning method (700) for a neural network, the learning method based on a quantization-sensitive technique, the quantization-sensitive technique implementing at least one quantization function, the quantization function enabling the quantization of at least one variable of a layer of neurons, the quantization function being characterized by a quantization precision (), the quantization precision being defined as the number of steps of the quantization function, the learning method being characterized in that the quantization precision of the quantization function is modified (770) progressively during the epochs of the learning method, to go from an initial quantization precision (2b+2) to a target quantization precision (2b),The initial and target quantization accuracies are predefined, with the initial quantization precision being strictly greater than the target quantization precision. Figure for the abstract: Figure 5.

Description

Title of the invention: Improved quantization-sensitive learning method for a neural network

[0001] The present invention relates to the field of neural networks, and, more particularly, to methods of quantifying neural networks.

[0002] A neural network is a particular type of artificial intelligence model, the purpose of which is to perform tasks such as image recognition, text translation, image generation...

[0003] A neural network is a model that takes as input an instance on which to perform the required task, and delivers as output a prediction on that instance.

[0004] A neural network is broken down into one or more layers of neurons.

[0005] A layer of neurons classically comprises an operating layer followed by an activation layer.

[0006] The operation layer can be, for example, a convolution layer or a "fully connected" layer. It performs a linear combination of input values.

[0007] An activation layer performs a generally non-linear function taking as input the result value of the operation layer and delivering as output one or more values ​​called activation values.

[0008] The activation values, which constitute the output values ​​of a layer of neurons, also constitute the input values ​​of the next layer of neurons.

[0009] Other layers may be present in a layer of neurons such as a normalization layer (“batch normalization” in English), or an averaging layer (“pooling” in English).

[0010] The organization of these different layers within a layer of neurons and of the different layers of neurons in relation to each other defines the architecture of the neural network. Numerous neural network architectures are known.

[0011] Some of the elementary operations performed by a layer are parametric, in the sense that they depend on one or more parameters, which constitute the parameters of the neural network and are the parameters to be learned. For example, the elementary operation performed by a convolutional layer uses weight parameters.

[0012] A neural network must therefore be trained in such a way as to learn the optimal values ​​of these different parameters in order to efficiently perform the task assigned to it.

[0013] Training is generally performed using data from a training set. This set may, for example, contain labeled instances. A labeled instance is an instance that is associated with an expected prediction. Comparing the prediction calculated by the neural network on an instance with the expected prediction associated with that instance allows, via the cost function (or "loss"), the values ​​of the neural network's parameters to be adjusted.

[0014] The training generally consists of carrying out several successive epochs, each comprising an inference phase ("Forward pass" in English) and a backpropagation phase ("backward pass" in English).

[0015] The result of the training is therefore a trained neural network (also called a "trained model" or "inference model"), that is, one whose various parameters are fixed at their optimal values. The trained neural network can then be used to perform the assigned task on new instances of input data.

[0016] However, for example for embedded applications, the trained neural network must be able to operate on a computing circuit whose memory and / or computing resources are reduced, or at least limited.

[0017] There is therefore a need for trained neural networks that have a restricted use of the resources of the computing circuit on which they are implemented.

[0018] To this end, the quantization of a neural network is the problem of reducing the number of bits on which the variables of the neural network are coded, whether they are the parameters of the trained model, or the values ​​that circulate through the trained model (in particular the activation values).

[0019] Quantization allows, for example, going from variables coded in floating precision (i.e. coded on 32 bits), to variables coded in reduced precision (i.e. coded on 8, 4, 2 or even 1 bit(s)).

[0020] Since a neural network comprises tens of thousands of variables, reducing the number of bits on which each parameter or activation value is encoded reduces the memory space required to store the trained neural network, as well as the intermediate computational data (such as activation values) during the execution of the trained neural network. Furthermore, this reduces processor usage, as the processor will use its integer arithmetic module rather than its floating-point arithmetic module. In particular, 1-bit word encoding precision corresponds to binary logic calculations, which are particularly simple for a processor to perform.

[0021] The challenge of quantization is then to maintain a performance close to the optimal performance, while reducing the memory and computational cost of the neural network, knowing that the optimal performance is obtained for this same network of neurons but with variables coded in floating-point precision trained for the task requested.

[0022] There are two main families of quantification techniques:

[0023] - Post-training quantification techniques (or “post-training” in (English): We consider a floating-point neural network already trained on the task to be performed, and we quantify its parameters to the desired target precision. Optionally, once the precision of the parameters has been reduced, we perform a retraining phase of the neural network at reduced precision, but with a training set smaller than that used for training the floating-point neural network.

[0024] - Quantization-sensitive training techniques - QAT ("Quantization- Aware Training): The neural network is trained directly with the goal of being quantized. Specifically, the inference phases of training are performed with quantized variables, while backpropagation operations are generally performed with floating-point gradients, often with a precision much higher than the quantization precision used for inference. Alternatively, during backpropagation phases, it is also possible to use quantized gradients with a similar or different precision to that used for the inference phases. Thus, a quantization error accumulates in the cost function of the neural network. Training the neural network then allows us to find the quantized parameters of the network that minimize the cost function. In summary, quantization-sensitive training techniques use quantized parameters and activation values ​​during training.Training is generally performed using the entire dataset from the training database. These techniques are less constrained (data access, unlimited training time), but are generally more difficult and much longer to execute and converge.

[0025] Among the most recent QAT techniques, we can mention:

[0026] - the scale-adjusted training technique - SAT, presented for example in the article by Qing Jin et al, "Towards Efficient Training for Neural Network Quantization", arXiv:1912.10207;

[0027] - the learned step size quantification technique - LSQ ("learned step size quantization"), presented for example in the article by Steven K. Esser et al., "Learned Step Size Quantization", arXiv: 1902.08153; and,

[0028] - the non-uniform to uniform quantification technique - N2UQ (“Nonuniform- to-Uniform Quantization"), presented for example in the article by Zechun Liu et al, "Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation", arXiv:2111.14826.

[0029] The object of the present invention is to propose an improvement to quantification-sensitive training techniques - QAT.

[0030] To this end, the invention relates to a learning method for a neural network, the learning method being based on a quantization-sensitive technique, the quantization-sensitive technique implementing at least one quantization function, the quantization function enabling the quantization of at least one variable of a layer of neurons, the quantization function being characterized by a quantization precision, the quantization precision being defined as the number of levels of the quantization function, the learning method being characterized in that the quantization precision of the quantization function is progressively modified during the epochs of the learning method, to go from an initial quantization precision to a target quantization precision, the initial and target quantization precisions being predefined,the initial quantification accuracy being strictly greater than the target quantification accuracy.

[0031] According to particular embodiments, the process comprises one or more of the following characteristics, taken individually or in all technically possible combinations:

[0032] - the neuron layer variable is a parameter of the neuron layer or an activation value of the neuronal layer.

[0033] - the quantization precision is modified to go from the precision of initial quantization to the target quantization precision via at least one intermediate quantization precision, the initial quantization precision being strictly greater than the intermediate quantization precision, which itself is strictly greater than the target quantization precision.

[0034] - the modification of the quantification accuracy during the epochs of the process The learning process implements a function to modify the precision of the quantification of the form:

[0035] p(k) = arr ( Ck^l) )

[0036] where j / k) is the quantization precision at epoch of rank k, j / 1) is the initial quantization precision used at epoch of rank 1, C is a decay factor strictly less than one, and arrQ is the rounding function.

[0037] - with the target quantization precision chosen to be equal to 2^, the precision of initial quantification is chosen equal to with A an integer greater than or equal to 1.

[0038] - the process includes the steps of: choosing the neural network architecture and a technique sensitive to the particular quantization; and, choice of a target quantization precision, an initial quantization precision, and a function of modifying the quantization precision allowing the quantization precision to be changed from one epoch to another in the learning process, then, for the epoch of rank k, k being an integer greater than one: carrying out an inference phase and a backpropagation phase at a current quantization precision p®; checking a criterion for the end of the learning process; and, if the check is negative, comparing the current quantization precision with the target quantization precision; and, if the current quantization precision is greater than the target quantization precision, modifying the current quantization precision for a subsequent epoch of rank k+1 using the quantization precision modification function.

[0039] - the quantization-sensitive technique is a scalar technique, defined as a technique in which each variable is quantified independently of other variables.

[0040] - the quantification-sensitive technique is chosen from: the technique adjusted scale training - S AT; the learned step size quantification technique - LSQ; and the non-uniform to uniform quantification technique - N2UQ.

[0041] - in a first part, the precision of quantification of the variable(s) of the The lowest neuronal layers of the neural network are progressively modified while maintaining a higher quantification accuracy for the variable(s) of the highest neuronal layers of the neural network, and then, in a second part, the quantification accuracy of the variable(s) of the highest neuronal layers is in turn progressively modified.

[0042] - the neural network is of the ResNet type.

[0043] - when the steps of the quantization function do not follow a distribution uniform, a position of each of the levels constitutes a parameter of the neural network.

[0044] The invention also relates to a method of configuring a target computing circuit with a trained neural network, comprising the steps of: providing a target computing circuit, whose operands are coded with a predefined coding precision; implementing a learning method in accordance with the preceding method by choosing a target precision equal to the coding precision, to obtain a trained neural network whose parameters are coded with the coding precision; and, transferring the trained neural network onto the target computing circuit.

[0045] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement a learning process conforming to the preceding process, in order to obtain a trained neural network in which at least one variable is quantified to a target quantization precision.

[0046] The invention also relates to a control system adapted to implement a learning process in accordance with the previous process in order to obtain a trained neural network in which at least one variable is quantified to a target quantification accuracy.

[0047] Preferably, the control system comprises: a computing circuit implementing a neural network and including storage means, the neural network calculating a prediction value from an instance value; at least one sensor configured to capture physical data as input and deliver instance values ​​as output to said neural network; at least one device receiving the prediction values ​​calculated by the neural network, characterized in that the neural network is a trained neural network resulting from the implementation of a learning process conforming to the preceding process, the parameters of the neural network learned as a result of the implementation of the learning process being stored in said storage means.

[0048] The invention also relates to a computer program product for implementing the present method.

[0049] The invention and its advantages will be better understood upon reading the following detailed description of a particular embodiment, given solely by way of non-limiting example, this description being made with reference to the accompanying drawings in which:

[0050] [Fig-1] Fig. 1 is a schematic representation of a neural network ResNet and the layers that make up a stage of this network;

[0051] [Fig.2] The [Fig.2] is a schematic representation of a convolution layer implementing a quantization-sensitive training technique;

[0052] [Fig.3] The [Fig.3] is a schematic representation of the implementation of the training of the convolution layer of the [Fig.2];

[0053] [Fig.4] The [Fig.4] is a schematic representation of an activation layer implementing a quantization-sensitive training technique of the SAT type;

[0054] [Fig.5] The [Fig.5] is a block representation of a method of training embodiment according to the invention;

[0055] [Fig.6] Fig.6 is a schematic representation of the implementation of training an activation layer by implementing a quantization-sensitive training technique of the N2UQ type; and,

[0056] [Fig.7] The [Fig.7] is a schematic representation of a computer system, such as a training computer for training an RN according to the method according to the invention or a target computer for executing the trained RN.

[0057] A neural network - RN (or model) is trained on a training computer, so as to obtain a trained neural network (or trained model).

[0058] The trained RN is then downloaded onto a target computer for use.

[0059] The training computer is generally not or only slightly limited in its computing and storage capabilities, unlike the target computer (such as, for example, a computer on board a vehicle, particularly an aircraft), intended to execute the trained RN.

[0060] A computer, such as the training computer or the target computer, can be a computer comprising computing means, such as a processor, and storage means, such as memory. The memory notably stores computer program instructions.

[0061] However, the invention is most relevant when the trained neural network is implemented by a target computer that corresponds to a coprocessor or computing circuit, particularly one optimized for neural computing, such as an arithmetic logic unit (ALU), a field-programmable gate array (FPGA), or the equivalent. Such computing circuits have minimal memory and computational capabilities.

[0062] An example of an implementation method of the learning process according to the present invention consists of the execution of a program by the training computer, the program being for example stored in a memory of the computer.

[0063] Alternatively, the learning process according to the present invention can be implemented using a circuit dedicated to neural computing, such as a neural accelerator, which allows learning to be performed without necessarily having to execute a computer program for all the operations of the learning process. Indeed, it is possible for all or some of the operations of the learning process to be carried out by dedicated hardware, such as a computer or a logic circuit implementing a finite state machine.

[0064] The learning method according to the invention aims to train a neural network -RN.

[0065] In the present description, the RN 1 to be trained is, for example, part of the ResNet architecture.

[0066] The ResNet architecture is known as such. It is described for example in the article Kaiming He et al, “Deep Residual Leaming for Image Recognition”, arXiv: 1512.03385.

[0067] The RN 1 is for example suitable for an image classification task.

[0068] Thus, the instance applied as input to RN 1 is an image, which has for example a resolution of three channels (Red, Green and Blue).

[0069] The prediction delivered at the output of RN 1 is then a probability of belonging to a class among a plurality of classes, each class corresponding to a pattern that we seek to recognize in the analyzed images.

[0070] The RN 1 consists of a plurality of floors.

[0071] It comprises an entrance floor 10, an exit floor 60, and a succession of intermediate floors 20, 30, 40 and 50.

[0072] The input stage includes, for example, a block 11 and the output stage includes, for example, a block 61.

[0073] The intermediate floors consist of a plurality of blocks.

[0074] By moving from one floor to another, the number of channels is doubled.

[0075] The blocks on the same floor have the same number of channels.

[0076] Thus, in the example of [Fig. 1]: block 20 comprises floors 21, 22 and 23, having for example 64 channels; block 30 has floors 31, 32, 33 and 34 having for example 128 channels; block 40 has floors 41, 42, 43, 44, 45, and 46, having for example 256 channels; and block 50 has floors 51, 52 and 53, having for example 512 channels.

[0077] The blocks of the intermediate floors have an identical structure. This structure is detailed on the right-hand side of [Fig. 1], for example for block 33.

[0078] The input value of the block is denoted Xj and the output value of the block is denoted Y'm.

[0079] A block comprises two successive layers of neurons, respectively 33_1 and 33_2.

[0080] The input value of the first layer of neurons 33_1 is the input value X} of the block.

[0081] The first layer of neurons 33_1 includes a first operation layer 100, which, from the input value of the first layer of neurons, Xj, allows obtaining an intermediate value of the first layer of neurons, Zj.

[0082] The first layer of neurons 33_1 then includes a first activation layer 200, which, from the intermediate value of the first layer of neurons, Zj, makes it possible to obtain an output value of the first layer of neurons, or activation value or activation, Yj.

[0083] The input value of the second layer of neurons 33_2 is denoted X 'j. It is equal to the activation value of the first layer of neurons Yj.

[0084] The second layer of neurons 33_2 includes a second operation layer 300, which, from the input value of the second layer of neurons, X1} allows obtaining an intermediate value of the second layer of neurons, Z'm.

[0085] The second layer of neurons 33_2 then includes a summation layer 400 performing the summation of the intermediate value of the second layer of neurons Z 'm and the input value of the block, X^ to obtain a summation output value, S 'm.

[0086] Finally, The second layer of neurons 33_2 includes a second activation layer 500, which, from the summation output value, S'm, allows obtaining an output value of the second layer of neurons, or activation value, Y'm.

[0087] The activation value of the second layer of neurons, Y'ni, is the output value of the block.

[0088] Fig. 2 explains the principle of a quantification-aware training technique (QAT) for the case of an operation layer, such as operation layer 100 of an intermediate block of RN 1.

[0089] The operation layer 100 includes a convolution block 105 and a weight quantification block 110.

[0090] At training time k (k being an integer between 1 and K), for the inference phase, the convolution block 105 performs the elementary convolution operation on the input value X to calculate the intermediate value Zj, an elementary operation which can, for example, take the generic form:

[0091] 2 • = ,Q^-X

[0092] where the are the weights of the convolution at epoch k. These are the parameters of the convolution layer.

[0093] The weights are quantified weights.

[0094] They are determined at the beginning of the inference phase of epoch k, from weights unquantified pyæ.

[0095] It is the weight quantization block 110 which performs this weight quantization operation by applying a weight quantization function, p(k).

[0096] The weight quantization function, p^Y is a step function, on an interval, preferably symmetric around the zero value and bounded by a maximum weight value.

[0097] The number of steps is the precision pW of the weight quantization function, also called quantization precision.

[0098] The spacing between two successive steps (or quantization steps) is preferably uniform but may not be. In particular, the steps may not be uniformly distributed within the interval if a scaling function allows a transition from a uniform distribution of steps to a non-uniform distribution. In other words, the quantization function used in the learning process according to the present invention must be parameterizable to set the number of steps and possibly the position of these steps, so that these two parameters can be varied as the learning process progresses (from one epoch to the next), as will be described in detail below with reference to [Fig. 5].It should be noted that when the steps of the quantization function do not follow a uniform distribution, a position of each of the steps constitutes a parameter of the neural network, which is learned either as such, or by adjusting the other parameters of the neural network, the function allowing the transition from a non-uniform distribution to a uniform distribution being absorbed by adjusting the parameters of the other layers.

[0099] Block 110 performs quantization with a quantization precision of weights. That is to say, the possible values ​​of a quantized weight pjFO are at number of nUd.

[0100] The quantization function applied by block 110 can optionally be parameterized by one or more weight quantization parameters qUO. These are parameters learned during training, just like the other parameters of RN 1, including the unquantized weight parameters ppW.

[0101] More specifically, in the case for example of the adjusted scale training technique - SAT, the quantization block 110 first performs a first clamping operation to constrain the value of the unquantized weights in an interval between 0 and 1.

[0102] This "projection" operation of unquantified weights makes it possible to obtain normalized weights. This operation consists of performing, for example, the following calculation:

[0103] (a) _ J / tanhU^y \ ÂJ 2 m^anhwÿl] )

[0104] The quantization block 110 then performs the actual quantization for determine the quantified weights.

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] [YES]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] In the case of the SAT technique, this operation consists of performing, for example, the following calculation: Where is the function defined by: q(x) = iarr(ax) In which the variable a is the quantization step, that is, the inverse of the number of quantization levels minus 1 (g = ct arrQ is the function rounded. This quantification operation is schematically illustrated in the upper part of Figure 3 representing the learning process 600. Applying the quantization function p® to a 2x2 matrix of unquantized weights ppdk) allows us to obtain a 2x2 matrix of quantized weights rfâ. We observe that the SAT technique is particularly simple since it does not use a quantization parameter for the weights q^). The quantified weights are calculated at the beginning of each epoch. This calculation is followed by the inference phase of the kth epoch of learning (step 610 in Figure 3). Labeled instances from a batch extracted from the training dataset are successively applied as input to RN 1 in order to calculate the corresponding predictions by RN 1 (which uses the parameter values ​​at iteration k, such as the weights zj(k) or the quantization parameters). ^ij At the end of the inference phase, epoch k of the learning process 600 continues with a backpropagation phase ("backward pass") 620. A loss function L (“loss”) is evaluated from the expected prediction and the obtained prediction. The backpropagation technique of the gradient of the loss function is implemented to calculate an update of each of the parameters of RN 1, including the quantization parameters of RN (where there are any). The gradient of the loss function L with respect to a variable x is denoted: thread. The property of chaining derivatives allows the gradient of the loss function with respect to the variable x to be calculated from the gradients of the loss function with respect to the intermediate variables between the output of the RN and the parameter x considered. Thus, the gradient is backpropagated from the highest layers of the RN, towards the lowest layers of the RN.

[0120]

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] More specifically, for the operation layer 100, as illustrated in Figure 2, backpropagation from the upper layers of RN 1 allows us to determine the gradient of the loss function with respect to the intermediate value Zj; dL / dZ j. Backpropagation of this gradient through block 105 with respect to the input value X2 (taking into account the variation dZj! dXj) allows us to go back to the gradient of the loss function with respect to the input value X± : dL,! dXf. Backpropagation of this gradient through block 105 with respect to each of the quantized weights q(L) (taking into account the variation .yd( / f^) allows us to trace back to the gradients of the loss function with respect to the quantized weights: dLjdd* Backpropagation of the gradient dL y dQ^) through block 110 with respect to the unquantized weights pyQd allows us to trace back to the gradient of the loss function with respect to the unquantized weights: dL / dW^' Finally, backpropagation of the gradient / dQ^ through block 110 with respect to possible quantization parameters qfâ allows us to trace back to the gradient of the loss function with respect to each quantization parameter of the weights: dL / d^- Backpropagation through the weight quantization block 110 is altered since the weight quantization function pW is not continuous-differentiable. According to the Straight Through Estimator (STE) technique, an STE estimator resulting from the smoothing of the quantization function of the weights plk) is used. For example, in the SAT technique, the derivative of the function d used in the definition of the weight quantization function is considered to be 1: ^[g(x)] = 1 Once the gradient of the loss function has been calculated with respect to a given parameter, it is used to update that parameter. As illustrated by step 630 in Figure 3, at the end of epoch k and the backpropagation phase of the gradient, the value of each unquantized weight ^y(k) is updated with the calculated gradient dL / dW^-'. This new value of the unquantized weights will be the one used for the next epoch k+1.

[0131] This training process with phases of inference and backpropagation is iterated for each epoch.

[0132] Training ends when a criterion is met. This could be a criterion relating to the number of epochs. For example, it is generally considered that around one hundred epochs (K=100) ensure convergence of the parameters of a ResNet. Of course, the convergence rate depends on the task, the training set, and the complexity of the model.

[0133] This may be a criterion on the value of the loss function and / or its gradient, such as a value below a threshold value.

[0134] Fig. 4 explains the principle of a QAT technique for the case of an activation layer, such as layer 200 of an intermediate stage of RN 1.

[0135] The activation layer 200 comprises an activation block 205 and an activation quantification block 210.

[0136] At training time k (k being an integer between 1 and K), for an inference phase, the activation block 205 performs the elementary activation operation on the intermediate value Zj to calculate the unquantized activation value Vj. This elementary operation consists of implementing an activation function, such as a non-linear function, like the ReLu function.

[0137] Then, the activation quantization block 210 performs the actual quantization, allowing the quantified activation value Yj to be obtained from the unquantified activation value Vj.

[0138] The activation quantization block 210 implements an activation quantization function, pUd.

[0139] This quantization function is characterized by a quantization precision, that is to say the number of quantization steps used in the function of quantification.

[0140] The quantization operation performed by block 210 can optionally be parameterized by one or more activation quantization parameters qW. These are parameters learned during training in the same way as the other parameters of the RN.

[0141] More precisely, in the case, for example, of the S AT technique, block 210 first performs a clamping operation to constrain the value of the unquantized activation value Vj to an interval between fi and a. The interval can be asymmetric, but it is preferably chosen to be symmetric with: / 3 = -Cf-

[0142] This "projection" operation of the unquantized activations makes it possible to obtain a normalized output Yj: A4'

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156] Vj-a\ + a] Block 210 performs the actual quantization to determine the quantized activation values ​​Yj. The following calculation is performed, for example: Where the function q^x) of the variable x is for example the same as that used by block 100. We observe that the S AT technique uses an activation quantification parameter, q <k\ en l’occurrence le paramètre a. For the inference phase of the learning process, labeled instances of a sample ("batch") extracted from the training base ("dataset") are successively applied as input to RN 1 in order to calculate the corresponding output (block 210 uses the value of the parameters at iteration k, such as parameter a). When all the sample data has been used, a backpropagation step is performed. A gap is calculated between the prediction calculated by RN 1 and the expected prediction for each instance in the sample. A loss function L (“loss”) is evaluated from the different deviations calculated for each instance of the sample. The backpropagation technique of the gradient of the loss function is implemented to calculate an update of each of the parameters of RN 1. More specifically, for the activation layer 200, as illustrated in Figure 4, backpropagation from the upper layers of RN 1 allows us to determine the gradient of the loss function with respect to the value Yj; dL / dYj. By backpropagation of the gradient dL / dYj through block 210, we calculate the gradient of the loss function with respect to the variable Vj: dL / dVj, then by backpropagation of this last gradient through block 205, we calculate the gradient of the loss function with respect to the value Zj: dL / dZj. Backpropagation of the gradient dL / dYj through block 210 with respect to the activation quantization parameters qik) allows us to trace back to the gradient of the loss function with respect to each activation quantization parameter: dL / dq^- Backpropagation through quantization block 210 must be altered since the activation quantization function p / k) is not continuous-differentiable.

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167] According to the Straight Through Estimator (STE) technique, an estimator resulting from the smoothing of the activation quantization function is used. For example, in the SAT technique, the derivative with respect to the variable x of the function ff is considered to be 1: and that the derivative of the function with respect to the quantization parameter a is equal to: for x < a for x > a This smoothing of the quantization function allows the variation of the loss function to be propagated not only to the input value of this block 210, but also to the activation quantization parameters. At the end of epoch k and the backpropagation procedure, the value of each activation quantization parameter is updated with the calculated gradient. This new value will be used for the next epoch k+1. This process of inference and backpropagation phases is iterated for each epoch, leading to the learning of the optimal value of the activation quantization parameters, the parameter a for the particular case of the SAT technique. The specific contribution of the present invention will now be presented in relation to the learning process shown in [Fig. 5]. This figure presents, in block form, a preferred embodiment of the process according to the invention. In general, the method according to the invention consists of modifying the quantization precision p(k) (pUd for an operation layer and p(k) for a layer activation (for example) of quantization functions, so that during the learning and iteration of epochs, the precision is gradually brought down from a higher initial quantization precision to a lower target quantization precision. It is important to emphasize that this refers to the modification of the quantization precision (that is, the number of quantization steps). This quantization precision is an integer. It can be changed by one or more units from one epoch to another. It is not the encoding precision of the variables, that is, the number of bits on which a variable (parameter or value) is encoded. In fact, during training, it is preferable to encode variables in floating-point precision (32 bits). However, since the goal of quantization is ultimately to obtain a trained neural network whose variables are encoded on a reduced number of bits, for example, b bits, the target quantization precision will preferably be set equal to the desired encoding precision, in this case, 2b.

[0168] Thus in a step 710 of process 700, the programmer makes a choice of the architecture of the neural network to be quantified and defines a particular quantification technique - QAT.

[0169] In a step 720, the programmer makes a choice on a target quantization precision. If, for example, the number of bits on which the programmer wishes to encode the variables of the learned model is b, the target quantization precision is chosen to be 2b.

[0170] The programmer also chooses the initial precision. It must be higher than the target precision. The initial precision can, for example, be defined as equal to 2b + A, with A an integer greater than or equal to 1, preferably equal to 2.

[0171] Finally, the programmer defines an evolution function defining the quantization precision at epoch k. For example, the evolution function is such that:

[0172] p(k)=arr( Ck2^) œ

[0173] where arrQ is the rounding function, C is a decay factor less than one, preferably close to one, for a slow decrease in quantification precision from one epoch to the next. C is, for example, chosen to be equal to 0.97.

[0174] In step 730, the epoch index k is initialized to the value unit.

[0175] For each epoch k, the process 700 provides the classic 740 steps of a learning process with implementation of a QAT quantification technique.

[0176] The inference and backpropagation phases of epoch k are carried out at the current quantization precision j / k. In other words, the precision of the quantization functions implemented at epoch k, whether by a convolution layer such as layer 100 or by an activation layer such as layer 200, is equal to

[0177] At the end of epoch k, step 750 verifies whether the learning end criterion is met. If so, process 700 is complete and the values ​​of the RN from the last epoch are the optimal values ​​that define the trained RN.

[0178] If not, process 700 continues with a step 760 consisting of comparing the current quantization precision j / k) with the target quantization precision, in our example 2^-

[0179] If the current quantization precision has not yet reached the target quantization precision, in step 770, the quantization precision for the next epoch ptk+1) is calculated using equation (1) above.

[0180] It should be noted that the rounding function does not systematically lead to a change in the quantization precision. This can remain constant over a a certain number of successive epochs before being reduced by one unit. The precision may therefore remain unchanged if the rounding function returns to the previous precision value.

[0181] Through successive iterations, the accuracy of the quantization functions gradually decreases during training.

[0182] The training continues by reducing the precision until the quantization precision is equal to the target quantization precision. That is to say, until the epoch Ko such that the precision pC^o) is equal to 2b.

[0183] Beyond this time, the training continues, but maintaining the quantification accuracy constant at the value of the target quantification accuracy (step 770 is therefore no longer carried out).

[0184] Thus, the training process is carried out with an initial precision of 2b + A during the first epoch k = 1, and is then progressively decreased until the target precision is reached, passing through various intermediate quantization precisions. Once the target quantization precision is reached, the precision of the quantization functions remains equal to the target precision until the end of the training.

[0185] This training method makes it possible to obtain a trained RN whose variables are quantized over 2b levels. They can therefore be coded on b bits.

[0186] For example, in order to configure a target computing circuit with a trained neural network, the following steps are carried out: provide a target computing circuit, whose operands are coded with a predefined coding precision; implement the previous learning process by choosing a target precision equal to the coding precision, to obtain a trained neural network whose parameters are coded with the coding precision; and transfer the trained neural network to the target computing circuit.

[0187]

[0188]

[0189] Fig. 6 is a schematic representation of the implementation of a 700 training process, for example on a convolution layer, by implementing a quantization-sensitive training technique of the N2UQ type. The difficulty with the N2UQ technique lies in the fact that the length 1 of each step of the weight quantization function pW is a quantization parameter q^\ However, if the precision of the quantization function is modified during training, it will be difficult to learn the optimal value of the step length at the same time. For the implementation of the present invention, two quantization functions are used: a function pW at the target precision p(0) and a function ptë) at the current precision of the epoch k (p®)-

[0190] For backpropagation phase 720, the STE estimator used in step 725 results from smoothing the function pW to the target accuracy ^0). The STE estimator is obtained by constructing a continuous curve, consisting of segments of constant slope, whose endpoints lie on two consecutive steps of the function p&\

[0191] This allows the gradient of the loss function to be calculated for the unquantized weights j clW^ ct c^cctucr 'a update of the unquantized weights (step 730).

[0192] This also allows the gradient of the loss function to be calculated with respect to the quantization parameters c?L / dq® and the quantization parameters q® to be updated (step 740), in particular the length7 of the weight quantization function steps p^i to the target accuracy p(0).

[0193] At the beginning of the inference phase 710, the quantization function of the weights p^ at the precision pUÔ is derived from the weight function p^ at the target precision p(°) (step

[0194] 745). It is essentially a matter of going from a fixed number of steps (for example four for the function pW) to another number of steps greater than or equal to (for example ten for the function pOd at epoch k), with the constraint of having steps spaced vertically by the same pitch a. Then the weight function p^ with precision pUO is used to determine (in step 705) the quantized weights r)0d from the unquantized weights vv ij

[0195] In this way, the learning of the quantification parameters is separated from the progressive modification of the quantification precision, which allows the learning process to converge.

[0196] It will be noted that with the quantization functions represented in [Fig.6], the quantized and unquantized weights are in the range between 0 and 2. The "clamping" operation is integrated into the weight quantization function.

[0197] The [Fig.7] is a schematic representation of a control system 800 according to an example of an embodiment of the present invention.

[0198] The control system 800 implements neural network 1 in its untrained version (and the control system then acts as a training computer) and / or in its trained version (and the control system then acts as a target computer). The learned version of neural network 1 results from the implementation of a learning method according to the present invention.

[0199] Physical input data, which may include x, y and z values, are captured by at least one 801 sensor. The 801 sensor is configured to deliver the captured values ​​as applied instances as input to neural network 1.

[0200] The neural network 1 is for example implemented in hardware form (“hardware”) and / or in software form (“software”) by a computing circuit 802, such as a central processor - CPU (“Central Processing Unit”) or a graphics processor - GPU (“Graphics Processing Unit”).

[0201] When the learned version of RN 1 is executed, the computing circuit 802 relies in particular on storage means 803 of the control system 800, which store all the parameters learned from RN 1. Each of these parameters is quantized to the target quantization precision defined during the implementation of the learning process according to the present invention. The storage means 803 then have the capacity to store operands defined by a number of bits equal to the target precision, for example 2b, with b an integer between 1 and 16.

[0202] When the unlearned version of RN 1 is executed, the computing circuit 802 performs the learning process according to the invention. In this case, there is no constraint on the size of the operants of the storage means 803, except that it must be greater than or equal to the target accuracy in order to store the value of the neural network parameters during their evaluation.

[0203] The execution of RN 1 allows a prediction to be obtained.

[0204] When the learned version of RN 1 is executed, this prediction is transmitted to an 804 device.

[0205] According to one or more embodiments, the 801 sensor is an optical sensor. For example, the 801 sensor is an optical sensor having a one-dimensional or two-dimensional pixel vector. The optical sensor is, for example, a visible light sensor, an infrared sensor, an ultrasonic sensor, or a depth sensor, such as a LiDAR (Light Detection and Ranging).

[0206] The physical input data is then one or more images captured by the optical sensor, and the RN 1 has the task of classifying the captured image(s) by performing image analysis. Such analysis can, for example, determine one or more actions to be performed by device 804.

[0207] Alternatively or in combination, the sensor 801 is a depth sensor, a temperature sensor, a microphone, or any other type of sensor. For example, the sensor 801 is a sensor for measuring the indoor and / or outdoor temperature of a building's heating / cooling system, including, for example, a heat pump as the primary energy source. In such cases, the device 804 is, for example, an activation circuit that turns the heating / cooling system on and off.

[0208] Alternatively or in combination, at least one device 804 is a display screen, a robotic system such as a robotic arm adapted to pull weeds or pick ripe fruit from a tree, an automatic steering or braking system for a vehicle, or an actuator, in particular electronic, for example configured to control the operation of one or more circuits, such as waking a circuit from a sleep mode, putting a circuit into a sleep mode, causing a circuit to generate text output, performs a data encoding or decoding operation, etc.

[0209] It should be noted that one can envisage a control system 800 implementing a trained version of the neural network at a low precision (for example in a low power consumption operating mode) and a trained version of the same neural network at a high precision (for example in a higher power consumption operating mode).

[0210] Alternative embodiments

[0211] The present invention can be applied to any RN architecture. The preceding description has shown its feasibility on convolutional and activation layers. These layers constitute the majority of RNs, regardless of their architecture.

[0212] While the progressive modification of the precision of the quantization functions has been presented more particularly in the case of the SAT technique, it can be shown mathematically that it applies to all types of QAT quantization techniques, provided that the latter is scalar.

[0213] By scalar, we mean a technique in which each variable of the RN is quantified independently of the other variables.

[0214] By uniform, we mean a technique in which the quantization functions subdivide the range of quantized values ​​into identical steps. Optionally, as mentioned above, a metric could be applied to modify the size of the quantization steps, but this metric must be predefined and not, for example, derived from statistics on the values ​​of the parameters to be quantified. This is referred to as a pseudo-uniform technique.

[0215] Alternatively, instead of using initial and final quantization accuracies that are powers of two, other values ​​for these quantization accuracies are possible. The only constraint is that the initial precision be chosen to be greater than the target precision in order to progressively constrain the model.

[0216] The quantization precision evolution function that allows the quantization precision to be modified from one epoch to another can be any function that gradually leads from the initial precision to the target precision. In particular, the evolution function could be such that, during a small number of intermediate epochs, the accuracy increases from one era to the next, in order to help for example with the convergence of the parameters and / or activation variables of the RN.

[0217] Preferably, the progressive modification of the quantization precision is implemented for all the quantization functions of all the layers of the RN simultaneously.

[0218] In alternative formulations, only some of the quantification functions could be subject to a progressive modification of their quantification precision.

[0219] Possibly, the target quantization precision of a quantization function could be different from that of another quantization function of the RN.

[0220] This would allow, for example, obtaining a trained RN with variables coded on different lengths.

[0221] Moreover, for the first layer of a neural network like ResNet, it is preferable to maintain a coding precision greater than or equal to 8 bits, even if the coding precision of the other layers of the RN is lower, in order to remain sensitive to the data of the input images.

[0222] In an advantageous embodiment of the learning method according to the invention, during the initial stages, only the accuracy of the quantization functions of the lowest layers of the RN is modified. The quantization accuracy for the highest layers of the RN is not modified to allow the variables (parameters and activation values) of these highest layers to adapt to the reduced accuracy of the variables in the lowest layers. Then, as training progresses, the accuracy of the quantization functions of the highest layers is also reduced. At the end of training, a trained RN is obtained in which all variables are quantized with the same target accuracy.

[0223] In another advantageous embodiment of the learning method according to the invention, the hardware-friendly mixed-precision quantization technique – HMQ (“Hardware-Friendly Mixed Precision Quantization”), presented, for example, in the article by Hai Victor Habi et al., “HMQ: Hardware-Friendly Mixed Precision Quantization Block for CNNs”, arXiv:2007.09952, is implemented to determine the optimal precision of each of the layers of the RN. Then, the learning method according to the invention is implemented to determine the quantized parameters, for one or more, or even each, of the layers of the RN at the corresponding target optimal precision. At the end of the training, a trained RN is obtained in which all parameters are quantized with the optimal precision relative to the layer to which the parameters belong.To do this, for the c layer of the RN, the programmer can choose the initial precision = 2bc+ A, with bc the target precision. specific to layer c, determined by the HMQ, and A common to all layers of the RN. As a variant, the value of A can be different for different layers of the neural network.

Claims

Demands

1. A computer-implemented learning method (700) for training a neural network (1) to obtain a trained neural network, the trained neural network being configured to be implemented by a computing circuit of a control system, the control system comprising at least one sensor and at least one device, the sensor being configured to capture physical data as input and deliver instance values ​​as output to said trained neural network, the trained neural network calculating a prediction value from an instance value delivered by the sensor, and the device receiving the prediction values ​​calculated by the trained neural network, the learning method being based on a quantization-sensitive technique,the quantization-sensitive technique implementing at least one quantization function, the quantization function allowing the quantization of at least one variable of a layer of neurons in the neural network, the quantization function being characterized by a quantization precision (p, nW), the quantization precision being defined as the number of steps of the quantization function, the learning process being characterized in that the quantization precision of the quantization function is modified (770) progressively during the epochs of the learning process, to go from an initial quantization precision to a target quantization precision, the initial and target quantization precisions being predefined, the initial quantization precision being strictly greater than the target quantization precision.

2. A method according to claim 1, wherein the neuron layer variable is a neuron layer parameter or a neuron layer activation value.

3. A method according to claim 1 or claim 2, wherein the quantification accuracy is modified to go from the initial quantification accuracy to the target quantification accuracy by passing through at least one intermediate quantification accuracy, the initial quantification accuracy being strictly greater than the intermediate quantification accuracy, which itself is strictly greater than the target quantification accuracy.

4. A method according to any one of the preceding claims, wherein the modification of the quantization precision during the epochs of the learning method implements a quantization precision modification function of the form: pW = arr(ck.pfl>) where p® is the quantization precision at epoch k, p® is the initial quantization precision used at epoch 1, C is a decay factor strictly less than one, and arrQ is the rounding function.

5. A method according to any one of the preceding claims, wherein, the target quantization precision being chosen to be 2^, the initial quantization precision is chosen to be equal to with A an integer greater than or equal to 1.

6. A method according to any one of the preceding claims, comprising the steps of: - choosing (710) the architecture of the neural network and a technique sensitive to particular quantization; and, - choosing (720) a target quantization precision, an initial quantization precision, and a quantization precision modification function enabling the quantization precision to be changed from one epoch to another in the learning process, and then, for epoch of rank k, k being an integer greater than one: - carrying out an inference phase and a backpropagation phase at a current quantization precision (p®); - checking (750) a criterion for the end of the learning process; and, if the check is negative, - comparing (760) the current quantization precision with the target precision;and, if the current quantization precision is greater than the target quantization precision, - modification of the current quantization precision for a subsequent epoch of rank k+1 using the quantization precision modification function.;

7. A method according to any one of the preceding claims, wherein the quantization-sensitive technique is a scalar technique, defined as a technique in which each variable is quantified independently of the other variables.

8. A method according to claim 7, wherein the quantization-sensitive technique is selected from: - the adjusted scale training technique - S AT; - the learned step size quantization technique - LSQ; and, - the non-uniform to uniform quantization technique - N2UQ.

9. A method according to any one of the preceding claims, wherein, in a first part, the quantification accuracy of the variable(s) of the lowest neuronal layers of the neural network is progressively modified while maintaining a higher quantification accuracy for the variable(s) of the highest neuronal layers of the neural network, and then, in a second part, the quantification accuracy of the variable(s) of the highest neuronal layers is in turn progressively modified.

10. A method according to any one of the preceding claims, wherein the neural network is of the ResNet type.

11. A method according to any one of the preceding claims, wherein, when the steps of the quantization function do not follow a uniform distribution, a position of each of the steps constitutes a parameter of the neural network.

12. A computer-implemented method for configuring a target computing circuit with a trained neural network, comprising the steps of: - providing a target computing circuit of the target control system, whose operands are coded with a predefined coding precision; - implementing a learning method (700) according to any one of claims 1 to 11 by choosing a target precision equal to the coding precision, to obtain a trained neural network whose parameters are coded with the coding precision; and, - transferring the trained neural network to the target computing circuit.

13. A computer program comprising software instructions which, when executed by a computer, implement a method according to any one of claims 1 to 11 in order to obtain a trained neural network in which at least one variable is quantized to a target quantization precision.

14. Control system (800) adapted to implement a method according to any one of claims 1 to 11 in order to obtain a trained neural network in which at least one variable is quantified to a target quantization accuracy.

15. Control system (800) comprising: - a computing circuit (802) implementing a neural network and including storage means (803), the neural network calculating a prediction value from an instance value; - at least one sensor (801) configured to capture physical data as input and deliver instance values ​​as output to said neural network; - at least one device (804) receiving the prediction values ​​calculated by the neural network, characterized in that the neural network is a trained neural network resulting from the implementation of a learning process according to any one of claims 1 to 11, the parameters of the trained neural network following the implementation of the learning process being stored in said storage means (803).

Images