METHOD FOR UPDATING AN ARTIFICIAL NEURAL NETWORK

Direct weight updates on hardware platforms using integer format calculations address memory and energy constraints, ensuring efficient adaptation of neural networks to new environments.

FR3116635B1Active Publication Date: 2024-11-29STMICROELECTRONICS (ROUSSET) SAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
FR2020012081
Authority / Receiving Office
FR · FR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-11-24
Publication Date
2024-11-29
Estimated Expiration
2040-11-24

AI Technical Summary

Technical Problem

Existing methods for updating neural network weights require significant memory resources and energy consumption, especially when performed on resource-constrained hardware platforms, and often result in suboptimal adaptation to the environment due to training on non-representative data.

Method used

A method for updating neural network weights directly on the hardware platform using integer format calculations, involving error gradient determination, weight correction, and quantification to maintain precision and reduce memory and energy consumption.

Benefits of technology

Enables efficient weight updates on resource-limited hardware without remote servers, reducing energy consumption and maintaining neural network performance by adapting to new environmental data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000021_0000
    Figure 00000021_0000
  • Figure 00000022_0000
    Figure 00000022_0000
Patent Text Reader

Abstract

According to one aspect, it is a method for updating an artificial neural network with initial weights stored in memory (MEM) in at least one integer format. This method comprises: - a determination by a processing unit (PU) of the error gradients at the output of the neural network layers, - a retrieval by the processing unit (PU) of the initial weights from memory (MEM), - an update of the initial weights by the processing unit (PU), including, for each initial weight, a first calculation of a corrected weight in the integer format of that initial weight, - a replacement by the processing unit (PU) of the value of the initial weights stored in memory (MEM) with the value of the corrected weights. Figure for the abbreviation: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Title of the invention: METHOD FOR UPDATING AN ARTIFICIAL NEURAL NETWORK

[0001] Embodiments and implementations relate to artificial neural networks, and more particularly to updating the weights of the layers of an artificial neural network after having implemented this neural network in a final hardware platform.

[0002] Artificial neural networks are used to perform given functions when executed. For example, one function of a neural network may be classification. Another function may be generating a signal from a signal received as input.

[0003] Artificial neural networks generally comprise a succession of layers of neurons.

[0004] Each layer takes as input data to which weights are applied and outputs output data after processing by activation functions of the neurons of said layer. This output data is transmitted to the next layer in the neural network.

[0005] Weights are configurable parameters to obtain good output data.

[0006] Neural networks can for example be implemented by final hardware platforms, such as microcontrollers integrated into connected objects.

[0007] Neural networks are generally trained during a learning phase before being integrated into the final hardware platform. The learning phase can be supervised or unsupervised.

[0008] The learning phase allows the weights of the neural network to be adjusted to obtain good output data from the neural network. To do this, the neural network can be executed by taking as input already classified data from a reference database. The weights are adapted according to the data obtained at the output of the neural network compared to expected data.

[0009] Furthermore, before being integrated into a hardware platform, the neural network may be quantized to be able to accelerate its execution and reduce its memory requirements. In particular, quantizing the neural network may consist of changing the format of the neural network data, such as the weights, which may be in a floating point format, to an integer format.

[0010] Quantizing the neural network allows for a lighter neural network. Such a neural network can be executed quickly by many platforms hardware, for example by microcontrollers.

[0011] Furthermore, in certain applications, it may be advantageous to provide updates to the weights of the layers after having implemented the neural network in the final hardware platform.

[0012] Indeed, for example, it may turn out that the neural network was trained from data that is not representative of the environment in which the neural network is implemented after having been implemented in the hardware platform.

[0013] Thus, the adaptation of the weights carried out during the learning phase may not be optimal for the environment in which the neural network is implemented.

[0014] Updating the weights of the neural network therefore makes it possible to adapt the neural network to the environment in which it is used.

[0015] The weights can be updated by a gradient backpropagation method, well known to those skilled in the art. This method uses a floating point weight format. In particular, the weights of the neural network represented in integers are converted to obtain a floating point weight format. Performing gradient backpropagation using floating point weights can require significant memory resources.

[0016] Thus, it may be necessary to update the weights of the neural network using a remote server when the final hardware platform does not have sufficient resources to perform this update.

[0017] The updated neural network is then quantized to obtain an integer weight format.

[0018] The updated quantized neural network is then transmitted to the final hardware platform, notably via the Internet network using wireless transmission.

[0019] Thus, the requested memory resources do not always allow the final hardware platform to perform the update itself. In addition, the energy consumption for transmitting the updated quantized neural network to the hardware platform is high for the hardware platform. However, the hardware platform generally has a limited amount of energy.

[0020] Furthermore, converting the weights into a floating point format to perform gradient backpropagation is memory expensive.

[0021] There is therefore a need to propose a method for updating a neural network directly by the final hardware platform.

[0022] According to one aspect, there is provided a method for updating an artificial neural network comprising initial weights stored in a memory according to at least one integer format, the method comprising:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] - a determination by a processing unit of the error gradients at the output of the layers of the neural network, - recovery by the processing unit of the initial weights in the memory, - an update of the initial weights by the processing unit comprising, for each initial weight, a first calculation of a corrected weight, in the integer format of this initial weight, according to the formula: Wa Ucorr = Wqii + rmmd (g q ij corr q ij \Scale(n) dw(l)ij / where is the value of the corrected weight, is the value of the initial weight, 2 is a q ij corr q ij learning rate, Scale(n) is a weight quantization step, V dE is dw(l)ij a sum of the error gradients with respect to the weights, and the round() function is a rounding up or down to an integer, - a replacement by the processing unit of the value of the initial weights stored in the memory by the value of the corrected weights. Thus, each weight is updated by adding to it a rounding of the sum of the error gradients with respect to the weights multiplied by a ratio between the learning rate 2 and the quantization step used to represent the initial weights. The calculation performed for updating the weights allows you to obtain weights directly in whole numbers. Such an update process therefore makes it possible to avoid using a floating point weight format. Therefore, the memory resources to perform the update are limited. The weight update process can thus be implemented directly by the final hardware platform that uses the neural network. For example, such a weight update process can be implemented directly by a microcontroller. This makes it possible to avoid updating the weights on a remote server. The weight update process therefore makes it possible to limit energy consumption, particularly when it is implemented by the final hardware platform. Moreover, such an update method is quick to implement because the calculation is performed directly on integer weights. A final hardware platform can thus regularly adapt the neural network from new data acquired in an environment in which the final hardware platform is located. Preferably, the sum of the error gradients with respect to the weights is calculated according to a so-called “batch” method well known to those skilled in the art.

[0033] In an advantageous embodiment, the updating of the initial weights is carried out for each layer of the neural network from a last layer to a first layer of the neural network.

[0034] In an advantageous embodiment, the rounding used to determine the value of the corrected weight is stochastic rounding.

[0035] Furthermore, the values ​​of the corrected weights may change after several weight updates, and may therefore become far from the values ​​of the neural network weights before the first update. In order to avoid saturation of the weights or a loss of precision of low-value weights, it is advantageous to re-quantize the corrected weights.

[0036] Thus, in an advantageous embodiment, the updating of the weights further comprises a quantification of the corrected weights.

[0037] Quantifying the corrected weights helps maintain good execution performance of the neural network.

[0038] In an advantageous embodiment, the quantification of a corrected weight of a layer of the neural network comprises a determination of a minimum weight and a maximum weight from among a set of the corrected weights of the layer, the set comprising the corrected weight to be quantified.

[0039] The set can be the set of weights of the layer when the layer is quantized uniformly or the set of weights of the same channel of the layer when the layer is quantized by channel.

[0040] In an advantageous embodiment, the quantification of the corrected weight further comprises: - a recovery by the processing unit of a value of a zero point associated with the initial weight stored in the memory, - an update by the processing unit of the value of a zero point associated with the corrected weight using the formula: , A ■ J >■ ( , / max (n)*minq-min (n)*maxq\ . Zp\n +1 j = int\clip\round ^_ min , mmq, maxqjj •> where Zp(n+1) is the updated zero point value, minq and maxq are respectively the minimum and maximum values ​​that the weights can take according to the weight format, min(n) = Scale(n).min(n)-Zp(n) and max(n) — Scale(n).max(n)-Zp(n), Zp(n) being the zero point value stored in memory, minq(n) being the minimum value of the corrected weights of the weight set, maxq(n) being the maximum value of the corrected weights of the weight set, - a replacement by the processing unit of the zero point value stored in the memory by the updated zero point value.

[0041] The clip() function is configured to maintain the values ​​of the expression max(n)*minq - min(n)*maxq \ in a range of values ​​[minq ; rounal max(n)-min(n) / maxq].

[0042] The int() function allows you to return an integer equivalent to the result of the expression / J max (n)*wiw-min (n)*maxq \ clip[ r ound(----- ,Lx

[0043] For example, when the weight format is unsigned and asymmetrical on 8 bits, the minimum value minq is equal to 0 and the maximum value maxq is equal to 255. When the weight format is signed and symmetrical on 8 bits the minimum value minq is equal to -127 and the maximum value maxq is equal to 127.

[0044] In an advantageous embodiment, the quantification of the corrected weight further comprises: - recovery by the processing unit of a value of a quantification step of the initial weight stored in the memory, - an update of the quantization step value by the processing unit according to the formula:

[0045]

[0046] Scale (n).(max (ny-min^(n)A, Scale (n + 1) - maxq - minq where Scale(n+1) is the value of the updated quantization step, minq(n) is the minimum value of the corrected weights of the weight set, maxq(n) is the maximum value of the corrected weights of the weight set, Scale(n) is the value of the quantization step stored in memory, minq and maxq are respectively the minimum and maximum values ​​that the weights can take, - a replacement by the processing unit of the value of the quantization step stored in the memory by the value of the updated quantization step. In an advantageous embodiment, quantifying the corrected weight comprises calculating a quantified value of the corrected weight. In particular, the quantified value of the corrected weight can be calculated using the formula:

[0047] go? con.(n +1) = round (maxq - minq) Scale(n) -Zp(n) Only (n) .(max (nj - min (n^ + Zp where w(l> + A is the quantized value of the corrected weight, / n\ is the q ij carr\ / q ij corr\ / value of the corrected weight obtained according to the first calculation, minq(n) is the minimum value of the corrected weights of the set of weights, maxq(n) is the maximum value of the weights corrected for the weight set, minq and maxq are respectively the minimum and maximum values ​​that the weights can take according to the weight format, Scale(n) is the old quantization step value stored in memory, Zp(n+1) is the updated zero point value and Zp(n) is the old zero point value stored in memory.

[0048] However, when the quantification is symmetrical, the quantized value of the corrected weight can be calculated, using the formula: \ / (2- v - 1 - 1 ) * _ .(n) w.. n + 1 - round--------- z xl ( h " ear A y ymax U min ( n ) |)] where fn + 11 is the quantized value of the corrected weight, Zn) is the q ij eorr\ / q ij corr\ / corrected weight value obtained according to the first calculation, minq(n) is the minimum value of the corrected weights of the weight set, maxq(n) is the maximum value of the corrected weights of the weight set, and N is the number of bits of the weight format (for example 8 bits).

[0049] Preferably, the values ​​that can be taken by the expression __1____for each possible value of the expression max dmax (n]\,\min (n)!) max JU min ( n ) |) are recorded in a lookup table stored in memory.

[0050] Furthermore, each layer of the neural network can be quantified uniformly, in which case we speak of per-layer quantization, or can be quantified differently for each channel of the layer, in which case we speak of per-channel quantization.

[0051] However, when updating the weights, it may be advantageous to adapt the type of quantization, by layer or by channel, of the layers of the neural network so as to improve the execution of the neural network.

[0052] In particular, during the learning phase, the neural network is quantized by layer or by channel depending on the learning results.

[0053] In general, channel quantization achieves better neural network accuracy for a slight increase in neural network execution time. This is because the execution of a channel-quantized layer is generally performed over a greater number of clock cycles than for a uniformly quantized layer.

[0054] However, in some cases channel quantization and layer quantization achieve equivalent accuracy. Thus, in this case, it may be advantageous to use layer quantization so as to allow a reduction of the execution time of the neural network.

[0055] Thus, in an advantageous embodiment, the method further comprises, for each layer, the following steps executed by the processing unit: - a calculation of a decision criterion, then - a comparison of the decision criterion with a threshold value, then - the quantification of the corrected weights which is carried out uniformly for each corrected weight of the layer or independently per channel of the layer according to a result of the comparison.

[0056] In an advantageous embodiment, the decision criterion is calculated according to the formula: _ ScaZeX / î + 1) “ min (Scale c ( n + 1 ) ) where Scale^n + 1) is a quantization step that can be defined for the entire layer and min(Scalejn + 1)) is the smallest quantization step among quantization steps that can be defined for each channel of the layer.

[0057] In an advantageous embodiment, for an initially uniformly quantified layer, if the decision criterion is greater than or equal to a threshold value between six and ten, then the quantization of the corrected weights is carried out according to a quantization per channel.

[0058] In an advantageous embodiment, for a layer initially quantized by channel, if the decision criterion is less than or equal to a threshold value greater than one and less than or equal to four, then the quantization of the corrected weights is carried out according to a uniform quantization for each corrected weight of the layer.

[0059] According to another aspect, there is provided a microcontroller comprising: - a memory configured to store initial weights of the layers of a neural network according to at least one given integer format, - a processing unit configured to perform: O a determination of the error gradients at the output of the layers of the neural network, O a recovery of the initial weights in the memory, O an update of the initial weights including, for each initial weight, a first calculation of a corrected weight, in the integer format of this initial weight, according to the formula: 0) (0 , > ( / 1 v dE V w; ; — w„ ; + round ------ Y , qi] corr qi] \Scale(n) dw(l)i) / where ^ / 9 is the value of the corrected weight, is the value of the initial weight, 2 is q ij corr q îj a learning rate, Scale(n) is a weight quantization step, V dE dw(l)ij is a sum of the error gradients with respect to the weights, and the round() function is a rounding up or down to an integer, O a replacement of the value of the initial weights stored in the memory by the value of the corrected weights.

[0060] Advantageously, the microcontroller therefore comprises a memory configured to store initial weights of the layers of a neural network according to at least one given integer format, and a processing unit configured to implement the method described previously.

[0061] According to another aspect, there is provided a computer program product comprising instructions which, when the program is executed by a processing unit, cause the latter to implement the method described above.

[0062] Other advantages and characteristics of the invention will appear on examining the detailed description of modes of implementation and embodiment, which are in no way limiting, and the appended drawings in which:

[0063] [fig.l]

[0064] [fig.2] schematically illustrate an embodiment and a method of implementation work of the invention.

[0065] [fig.l] illustrates an MCT microcontroller used as the final hardware platform integrating a neural network.

[0066] The neural network is defined by parameters well known to those skilled in the art, such as weights.

[0067] The neural network comprises an input layer, an output layer and at least one hidden layer between the input layer and the output layer.

[0068] Each layer may comprise at least one channel. Each channel comprises at least one weight.

[0069] The weights of each layer are integers defined according to one or more formats. In particular, each weight is defined according to its quantized value, a quantization step and a zero point.

[0070] In particular, a weight is expressed according to the following formula: .y,,, x ( - zp w ) , where sw is the quantization step, qw is the quantized value of the weight and zpw is the zero point of quantization.

[0071] Each layer can be quantized uniformly or by channel. When the layer is quantized uniformly, all the weights of a layer have the same format. When the layer is quantized by channel, the weights of the same channel have the same format. However, the weight formats of different channels may not be the same. Each channel is therefore quantized independently of the other channels of the layer.

[0072] The values ​​of the weights of the neural network as well as their quantification were defined after a learning phase carried out before the neural network is integrated into the MCT microcontroller. The learning phase can be supervised or unsupervised. The learning phase allows the weights of the neural network to be adjusted. To do this, the neural network can, for example, be executed by taking as input already classified data from a reference database.

[0073] In particular, the MCT microcontroller comprises a memory MEM in which the parameters of the neural network are stored, in particular the weights of the layers of the neural network, i.e. the quantized values ​​of the weights, their quantization steps and their zero points. The memory MEM is a non-volatile memory.

[0074] The microcontroller MCT also comprises a processing unit UT. The processing unit UT is a processor.

[0075] The processing unit UT is configured to be able to execute the neural network from data received as input from the neural network.

[0076] The execution of the neural network makes it possible to deliver output data based on the data received as input from the neural network.

[0077] For example, a neural network can indicate as output a classification of the data received as input, or generate an output signal from a signal received as input to the neural network.

[0078] The neural network may be implemented in an environment in which the MCT microcontroller is placed. This environment may be different from the environment in which the neural network was trained.

[0079] In order to adapt the neural network to the environment in which it is implemented, the processing unit UT is also configured to update the weights of the neural network by implementing the method described below in relation to [fig.2]. In particular, the microcontroller MCT has in the memory MEM a computer program product comprising instructions which, when executed by the processing unit UT, lead the latter to implement the method described below.

[0080] In particular, the method comprises a step 20 in which the processing unit UT implements the neural network several times with different data. The processing unit UT calculates a total error according to a so-called “batch” method well known to those skilled in the art from the output results of the neural network following the different implementations of the latter.

[0081] The method then comprises a step 21 in which the processing unit UT calculates error gradients at the output of the layers of the neural network. In particular, the processing unit UT calculates the error gradients using a gradient backpropagation method.

[0082] The method then comprises a succession of steps 22 to 37 which is carried out for

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] each layer from the output layer of the neural network to the input layer of the neural network. In particular in step 22, the processing unit UT recovers the weights, called initial weights, of the layer stored in the memory MEM. Then, in step 23, the processing unit UT updates the weights of the layer. In particular, for each weight of the layer, the processing unit UT performs a first calculation of a corrected weight in the integer format of the initial weight stored in the memory MEM. In particular, the UT processing unit calculates each corrected weight using the following formula: (O CD . , ( V dE 3 qi) corr q ij \Scale(n) dw(l)ij / where is the value of the corrected weight, is the value of the initial weight, 2 is qij corr q ij a learning rate, Scale(n) is a quantization step of the weight, . dE dw(l)ij is a sum of the error gradients with respect to the weights, and the round() function is a rounding up or down to an integer. Thus, each weight is updated by adding to it a rounding of the sum of the error gradients with respect to the weights multiplied by a ratio between the learning rate 2 and the quantization step used to represent the weights. Preferably, the rounding used to determine the weight value is stochastic rounding. The method then comprises steps 24 to 36 for adapting the quantification of the corrected weights of the layer to the values ​​of these corrected weights. In particular, in step 24, the processing unit UT determines a minimum weight and a maximum weight from at least one set of the corrected weights of the layer. More specifically, the UT processing unit determines the minimum weight and the maximum weight of the set of corrected weights of the layer. Additionally, the UT processing unit determines the minimum weight and maximum weight of the corrected weights of each channel of the layer. Then, in step 25, the processing unit UT calculates quantization steps of the corrected weights of the layer. In particular, the UT processing unit calculates a quantization step for the set of corrected weights of the layer using the minimum weight and the maximum weight of the set of corrected weights of the layer. To do this, the UT processing unit applies the following formula: Scale (n).(max ^ny-min^ maxq - minq Scale, ( n + 1 ) = Scale (n + 1) where minqi(n) is the minimum value of the corrected weights of the layer, maxqi(n) is the maximum value of the corrected weights of the layer, Scale(n) is the value of the quantization step of the initial weights of the layer, minq and maxq are respectively the minimum and maximum values ​​that the weights can take according to the weight format.

[0095] The Scale(n) value is retrieved from the MEM memory by the processing unit UT.

[0096] For example, when the weight format is unsigned and asymmetrical on 8 bits, the minimum value minq is equal to 0 and the maximum value maxq is equal to 255. When the weight format is signed and symmetrical on 8 bits, the minimum value minq is equal to -127 and the maximum value maxq is equal to 127.

[0097] Furthermore, for each channel of the layer, the processing unit UT calculates a weight quantization step for this channel using the minimum weight and the maximum weight of this channel. To do this, the processing unit UT applies the formula: Scale (n).{max {nymin^ maxq - minq where minqc(n) is the minimum value of the corrected channel weights, maxqc(n) is the maximum value of the channel weights, Scale(n) is the quantization step value of the initial channel weights, minq and maxq are respectively the minimum and maximum values ​​that the weights can take.

[0098] Then, in step 26, the processing unit UT calculates a decision criterion in order to know whether it is preferable to quantize the layer uniformly or by channel. The decision criterion is calculated according to the formula: _ Scale +1) , where Scale+ 1) is the calculated quantization step “ min (5(+ / 5^ / / + for the whole layer and ni in {Scale c(n + 1)) is the smallest quantization step of the quantization steps calculated for the different channels.

[0099] Then, in step 27, the processing unit UT determines whether the layer is currently quantized uniformly or by channel.

[0100] If the layer is initially uniformly quantized, then the processing unit UT performs a test in step 28. In this test, the processing unit UT compares the value of the decision criterion to an integer value X included in the interval [6; 10]. If the decision criterion is greater than or equal to the value X, then the processing unit UT determines that it is preferable to quantize the layer by channel for the new corrected weights. Otherwise, the processing unit UT considers that it is preferable to maintain uniform quantization for the layer.

[0101] Furthermore, if the layer is initially quantized by channel, then the processing unit UT performs a test in step 29. In this test, the processing unit UT compares the value of the decision criterion to an integer value Y included in the interval ]1; 4]. If the decision criterion is less than or equal to the value Y, then the processing unit

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] UT determines that it is best to quantize the layer uniformly for the new corrected weights. Otherwise, the UT processing unit considers it best to maintain per-channel quantization. The intervals chosen for the integer values ​​X and Y are defined from experiments. Then, if the layer is to be uniformly quantized, then the processing unit UT performs uniform quantization of the layer weights in steps 30 to 33. In particular, in step 30, the processing unit UT updates the value of the zero point stored in the memory MEM. To do this, the value of the zero point Zp(n) is retrieved from the memory MEM by the processing unit UT. Then, the processing unit UT calculates the new zero point using the following formula: -, ( A ■ I ,■ I max (n)*minq~ min (n)*maxq j . \j Zp n + 1 = inn clip\ round -----\ „ / <----- , mina, maxq • \ / \ \ \ max (n)-mm (n) / •> where Zp(n+1) is the updated zero point value, minq and maxq are respectively the minimum and maximum values ​​that the weights can take, min(n) - Scale(n).min,.(n)-Zp(n) and max(n) = Scale(H).max,(n)-Zp(n), Zp(n) being the value of the zero point stored in the MEM memory, minqi(n) being the minimum value of the corrected weights of the layer, maxqi(n) being the maximum value of the corrected weights of the layer. Then, in step 31, the processing unit UT updates the value of the quantization step by the processing unit UT according to the following formula: Scale (n) .(max (n)-min ________________ '_________________s!____________________' maxq - minq where Scalei(n+1) is the value of the updated quantization step of the layer weights, minqi(n) is the minimum value of the corrected weights of the layer, maxqi(n) is the maximum value of the corrected weights of the layer, Scale(n) is the value of the quantization step of the initial weights of the layer, minq and maxq are respectively the minimum and maximum values ​​that the weights can take. It should be noted that this quantization step Scalei(n+1) has already been calculated in step 25. Thus, the processing unit UT does not necessarily perform the calculation if the value Scalei(n+1) has been previously stored in the memory MEM. Then, in step 32, the processing unit UT quantifies the corrected weight value again. In particular, the quantified value of the corrected weight can be calculated using the formula: Scale, ( n + 1 ) = (0 iv !. q ij corr (n + 1) - round (maxq - minq). Scale(n) -Zp(n) Scale (n).(max (nj - min (rty + Zp n + 1 where w(l> (n + ] | is the quantized value of the corrected weight, in\ is the value q îj corry / q ij corry / of the corrected weight obtained according to the first calculation, minq(n) is the minimum value of the corrected weights of the set of weights, maxq(n) is the maximum value of the corrected weights of the set of weights, minq and maxq are respectively the minimum and maximum values ​​that the weights can take according to the weight format, Scale(n) is the old value of the quantization step stored in the memory, Zp(n+1) is the updated value of the zero point and Zp(n) is the old value of the zero point stored in the memory.

[0111] However, when the quantization is symmetrical, the processing unit UT can use the formula: ... / \ / (2^-1)^ / °... y w { \. n + 1 = round---- n \h where + ]j is the corrected quantized value of the weight, / n\ is the q ij corry / q ij corry / corrected weight value obtained according to the first calculation, minqi(n) is the minimum value of the corrected weights of the weight set, maxqi(n) is the maximum value of the corrected weights of the weight set, and N is the number of bits of the weight format (for example 8 bits).

[0112] Preferably, the values ​​that can be taken by the expression _________1_______________for each possible value of the expression max | min l ( n ) j) max min^, ( n ) [) are recorded in a lookup table stored in MEM memory.

[0113] For example, the values ​​that can be taken by the expression max min^ ( n ) |) are generally close to 127 when the value of the weights is represented using an eight-bit symmetric quantization. Indeed, the corrected values ​​of the weights are generally close to the weight values ​​before their update. The values ​​that can be taken by the expression ______________ 1_______________for these values ​​are therefore precalculated then max min jn ) |) stored in the lookup table.

[0114] Then, the values ​​of the weights, the quantization step and the zero point stored in the memory MEM are then replaced in step 33 by the newly calculated.

[0115] Furthermore, if the layer must be quantized by channel, then the processing unit UT performs a quantization of the weights for each channel of the layer in steps 34 and 37.

[0116] In particular, in step 34, the processing unit UT updates the value of the zero point stored in the memory MEM. To do this, the value of the zero point Zp(n) is retrieved from the memory MEM by the processing unit UT. Then, the processing unit UT calculates the new zero point using the following formula:

[0117] 7 z ■ J i- IJ max (n)*minq - min (n)*maxq\ . \\ Zp (n + 1) = clip round ----un-min' mm ^ \ \ \ 41.x\H} x 11X11 \ / 1 l / / / Scale c ( n + 1 ) where Zp(n+1) is the updated zero point value, minq and maxq are respectively the minimum and maximum values ​​that the weights can take, min(n) = Scale(n) .min^(n)-Zp(n) and max(n) — Scale(n).max(n)-Zp(n), where Zp(n) is the zero point value stored in MEM memory, minqc(n) is the minimum value of the channel's corrected weights, maxqc(n) is the maximum value of the channel's weights.

[0118] Then, in step 35, the processing unit UT updates the value of the quantization step by the processing unit UT according to the following formula: Scale (n).(max (n)-min maxq - minq where Scalec(n+1) is the value of the updated quantization step of the channel weights, minqc(n) is the minimum value of the corrected channel weights, maxqc(n) is the maximum value of the channel weights, Scale(n) is the value of the quantization step of the initial channel weights, minq and maxq are respectively the minimum and maximum values ​​that the weights can take.

[0119] It should be noted that this quantization step Scalec(n+1) has already been calculated in step 25. Thus, the processing unit UT does not necessarily perform the calculation if the value Scalec(n+1) has been stored in the memory MEM previously.

[0120] Then, in step 36, the processing unit UT quantifies again the value of the corrected weights.

[0121] In particular, the quantified value of the corrected weight can be calculated using the formula:

[0122] m / corr(n+1) = round(maxq-minq) Scale(n) -Zp(n) Scale (n).(max (nj - min (n^ where In + । j is the quantized value of the corrected weight, / n\ is the q ij corry / q ij corry ) corrected weight value obtained according to the first calculation, minq(n) is the minimum value of the corrected weights of the set of weights, maxq(n) is the maximum value of the corrected weights of the set of weights, minq and maxq are respectively the minimum and maximum values ​​that the weights can take according to the weight format, Scale(n) is the old value of the quantization step stored in the memory, Zp(n+1) is the updated zero point value and Zp(n) is the old zero point value stored in the memory.

[0123] However, when the quantization is symmetrical, the processing unit UT can use the formula: U0 q ij corr n+ 1 = round--- [max where + n is the corrected quantized weight value, / n ) is the q ij corry / q ij corry / corrected weight value obtained according to the first calculation, minq(n) is the minimum value of the corrected weights of the weight set, maxq(n) is the maximum value of the corrected weights of the weight set, and N is the number of bits of the weight format (for example 8 bits).

[0124] As previously, the values ​​that can be taken by the expression _______________1_______________for each possible value of the expression max I) max min^c(n ) |) are recorded in a lookup table stored in MEM memory.

[0125] Then, the values ​​of the weights, the quantization step and the zero point stored in the memory MEM are then replaced in step 37 by the newly calculated values.

[0126] After having carried out steps 22 to 37 for a given layer of the neural network, the processing unit repeats these steps for the layer which directly precedes said given layer in the neural network.

[0127] The calculation carried out for updating the weights makes it possible to obtain weights directly in whole numbers.

[0128] Such an update method therefore makes it possible to avoid using a floating point weight format.

[0129] As a result, the memory resources for performing the update are limited.

[0130] The weight updating method can thus be implemented directly by the final hardware platform that uses the neural network. For example, such a weight update method can be implemented directly by an MCT microcontroller.

[0131] It is thus possible to avoid updating the weights on a remote server.

[0132] The weight updating method therefore makes it possible to limit energy consumption, particularly when it is implemented by the final hardware platform.

[0133] Furthermore, such an update method is quick to implement because the calculation is carried out directly on whole weights.

[0134] A final hardware platform can thus regularly adapt the neural network from new data acquired in an environment in which the final hardware platform is located.

[0135] Adapting the quantization of the corrected weights makes it possible to maintain good execution performance of the neural network.

Claims

Claims

1. Method for updating an artificial neural network comprising initial weights stored in a memory (MEM) according to at least one integer format, method comprising: - a determination by a processing unit (UT) of the error gradients at the output of the layers of the neural network, - a recovery by the processing unit (UT) of the initial weights in the memory (MEM), - an update of the initial weights by the processing unit (UT) comprising, for each initial weight, a first calculation of a corrected weight, in the integer format of this initial weight, according to the formula: J corr = l + round (scaieW Z dw(i)ij), where is the value of the corrected weight, is the value of the initial weight q ij corr q ij, 2 is a learning rate, Scale(n) is a quantization step of the weight, V dE is a sum of the error gradients dw(l)ij with respect to the weights,and the round() function is a rounding to a higher or lower integer, - a replacement by the processing unit (UT) of the value of the initial weights stored in the memory (MEM) by the value of the corrected weights.,

2. The method of claim 1, wherein the updating of the initial weights is performed for each layer of the neural network from a last layer to a first layer of the neural network.

3. A method according to any one of claims 1 or 2, wherein the rounding used to determine the value of the corrected weight is stochastic rounding.

4. Method according to one of claims 1 to 3, in which the updating of the weights further comprises a quantification of the corrected weights.

5. The method of claim 4, wherein quantizing a corrected weight of a layer of the neural network comprises determining a minimum weight and a maximum weight from a set of the corrected weights of the layer, the set comprising the corrected weight to be quantized.

6. The method of claim 5, wherein quantifying the corrected weight further comprises: - a recovery by the processing unit (UT) of a value of a zero point associated with the initial weight stored in the memory (MEM), - an update by the processing unit (UT) of the value of a zero point associated with the corrected weight using the formula: „ / . 7 / , / niax (n)*minq- min (n)*maxq\ . \\ ZP\n+ l^mt[chp(roUnd] ), mmq, maxq}} •> where Zp(n+1) is the updated value of the zero point, minq and maxq being respectively the minimum and maximum values ​​that the weights can take, min (n) = Scale ( n ) .min ( n)-Zp(n) and max (n) = Scale ( n ) .max. (n)-Zp(n), Zp(n) being the value of the zero point stored in the memory (MEM), minq(n) is the minimum value of the weights corrected from the set of weights, maxq(n) is the maximum value of the corrected weights from the set of weights, - a replacement by the processing unit (UT) of the zero point value stored in the memory (MEM) by the updated zero point value.

7. Method according to claim 6, in which the quantification of the corrected weight further comprises: - a recovery by the processing unit (UT) of a value of a quantization step of the initial weight stored in the memory (MEM), - an update of the value of the quantization step by the processing unit (UT) according to the formula: Scale ( n ) .(max (n)-min((n)), Scale ( n+ 1 ) - maxq - minq where Scale(n+1) is the value of the updated quantization step, minq(n) is the minimum value of the corrected weights of the set of weights, maxq (n) is the maximum value of the corrected weights of the set of weights, Scale(n) being the value of the quantization step stored in the memory (MEM), minq and maxq being respectively the minimum and maximum values ​​that the weights can take, - a replacement by the processing unit (UT) of the value of the quantization step stored in memory (MEM) by the value of the updated quantization step.

8. The method of claim 7, wherein quantifying the corrected weight comprises calculating a quantized value of the corrected weight, using the formula: A \ / (2” 1 - 1) * (n) V iv l. n + 1 = round----- g tj con-y j ।max Hmax (nj ,| min {n)\)j where fn + ] 1 is the quantized value of the corrected weight, f ni is the value of the corrected weight obtained according to the first q ij / calculation, minq(n) is the minimum value of the corrected weights of the set of weights, maxq(n) is the maximum value of the corrected weights of the set of weights.

9. Method according to one of claims 4 to 8, further comprising, for each layer, the following steps executed by the processing unit: - a calculation of a decision criterion, then - a comparison of the decision criterion with a threshold value, then - the quantification of the corrected weights which is carried out uniformly for each corrected weight of the layer or independently by channel of the layer according to a result of the comparison.

10. The method of claim 9, wherein the decision criterion is calculated according to the formula: _ Scale^n + 1 ) ~ min (Scalec ( n + 1 ) ) where Scale^n + 1) is a quantization step that can be defined for the entire layer and min (Scalec(n +1)) is a smallest quantization step among quantization steps that can be defined for each channel of the layer.

11. The method of claim 10, wherein, for an initially uniformly quantized layer, if the decision criterion is greater than or equal to a threshold value between six and ten, then the quantization of the corrected weights is performed according to a channel quantization.

12. A method according to any one of claims 10 or 11, wherein, for a layer initially quantized per channel, if the decision criterion is less than or equal to a threshold value greater than one and less than or equal to four, then the quantization of the corrected weights is carried out according to a uniform quantization for each corrected weight in the layer.

13. Microcontroller comprising: - a memory (MEM) configured to store initial weights of the layers of a neural network according to at least one integer format given, - a processing unit (TU) configured to perform: O a determination of the error gradients at the output of the layers of the neural network, O a recovery of the initial weights in the memory (MEM), O an update of the initial weights including, for each initial weight, a first calculation of a corrected weight, in the integer format of this initial weight, according to the formula: WaUcorr = Waü + round (cq ij corr q ij \Scale(n) dw(l)ij / where is the value of the corrected weight, is the value of the weight q ij corr q ij initial, 2 is a learning rate, Scale(n) is a weight quantization step, V dE is a sum of error gradients dw(l)ij relative to the weights, and the round() function is a rounding up or down to an integer, O a replacement of the value of the initial weights stored in memory (MEM) by the value of the corrected weights.

14. Microcontroller, comprising a memory (MEM) configured to store initial weights of the layers of a neural network according to at least one given integer format, and a processing unit (UT) configured to implement the method according to one of claims 1 to 12.

15. Computer program product, comprising instructions which, when the program is executed by a processing unit (UT), cause the latter to implement the method according to one of claims 1 to 12.