Method and device for optimizing at least one artificial neural network and associated computer program
Patent Information
- Application Number
- FR2023009423
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-03-14
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method and device for optimizing at least one artificial neural network and associated computer program
[0001] The present invention relates to a method for optimizing at least one artificial neural network, said method being implemented during the training phase of said at least one artificial neural network, said neural network comprising an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer, said method being implemented by an electronic device for optimizing at least one artificial neural network.
[0002] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement such a method for optimizing at least one artificial neural network.
[0003] The invention also relates to said device for optimizing at least one artificial neural network.
[0004] The present invention relates to the field of artificial intelligence and embedded systems and relates more specifically to the optimization of artificial neural networks. An artificial neural network is classically a set of elementary computing units, called neurons, connected together and whose weights, floating numbers, are suitable for being learned classically by a predetermined algorithm in order to optimize a merit score (i.e. cost of a cost function) from a training data set.
[0005] Generally speaking, the larger the artificial neural network is in terms of size (number of parameters, depth, etc.), the more it will be capable of performing complex and high value-added functions. However, the complexity of networks is currently evolving much faster than computing power, which poses many challenges, particularly for optimizing inference on embedded systems (e.g. embedded targets) or not, or for optimizing energy consumption more generally. Indeed, the neural network models currently used are generally oversized to facilitate training due to redundant elements.
[0006] Optimizing neural networks and reducing their hardware footprint by removing redundant elements, which amounts to compression with more or less loss, is both interesting for embedded targets whose computational budget is reduced and also for remote servers in order to increase their energy performance in particular.
[0007] To do this, many solutions have been developed so far such as structured or unstructured pruning, linear or nonlinear quantization, low-rank approximation via layer decomposition, knowledge distillation, etc.
[0008] Each of these solutions works well separately, but there is often still redundancy.
[0009] A sequential approach to these solutions is also likely to increase the risk of only a local optimum.
[0010] The application of different solutions also requires determining, in particular manually or via a search “meta-algorithm”, a large number of hyperparameters without any guarantee of relevant convergence or in a tedious manner in most cases. Indeed, each method adds different hyperparameters to be selected, and these hyperparameters have an interaction that is difficult to predict, which complicates training.
[0011] The aim of the invention is then to propose a solution for optimizing artificial neural networks which makes it possible to more effectively reduce redundancy while avoiding the pitfalls of the aforementioned solutions.
[0012] To this end, the invention relates to a method for optimizing at least one artificial neural network, said method being implemented during the training phase of said at least one artificial neural network, said neural network comprising an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer, said method being implemented by an electronic device for optimizing at least one artificial neural network, and comprising the following successive steps:
[0013] - tensor decomposition of said at least one artificial neural network and in training said at least one decomposed artificial neural network;
[0014] - quantification of at least one element resulting from said decomposition using a bit size, said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function;
[0015] - pruning of said at least one artificial neural network resulting from said quantification by removing at least one element associated with a bit size less than a predetermined bit size threshold;
[0016] - recomposition of at least one layer of said at least one neural network ar pruned artificial neural networks when, after said pruning, the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer and training of said at least one recomposed artificial neural network;
[0017] - obtaining said at least one optimized neural network by removing any previously applied quantification.
[0018] Thus, the present invention proposes to cleverly combine three state-of-the-art compression solutions and completely unifying them to optimize them jointly during neural network training (finetuning).
[0019] More specifically, the present invention has a stronger compression potential than that of the current state of the art which generally implements one or even two of the solutions among the three cleverly combined according to the present invention.
[0020] More precisely, the order of combination of the steps of the method according to the present invention is essential because it allows firstly to consider the low rank approximation (i.e. the rank selection from the English low rank approximation) as structured pruning (from the English pruning) after the implementation of the first decomposition step, then to consider the structured pruning as a particular case of a quantification, for example linear (but not necessarily), with low granularity.
[0021] Under the unique prism of said low-granularity quantization, it is then proposed according to the present invention to unify the three quasi-orthogonal solutions, namely the low-rank approximation by decomposition, the quantization and the pruning. Indeed, it is the low-granularity quantization which will impact the merit function (i.e. cost function) and consequently define the associated compression rate of the artificial neural network.
[0022] According to other advantageous aspects of the invention, the method for optimizing at least one artificial neural network comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:
[0023] - the method (further comprising, between said recomposition step and said step of obtaining, a step of standardizing each layer of said at least one recomposed artificial neural network retrained by determining a single bit size associated with each layer and adjusting the weights of said at least one artificial neural network standardized by retraining;
[0024] - each bit size applied during the quantization step is a number decimal;
[0025] - said quantification is affine according to the formula: with x ER, ae R+> and such that Q b : x ■-> x t} = ax clamp^ round( v h (b}) + A fi g R avec c ^ am P a clipping function between two predetermined limits and Th(b), and round a function returning a rounded value;
[0026] -T^ = -2W+ietr / ; (b) =2W-1;
[0027] - the predetermined cost function is a regularized cost function expressed in the following form: LOS S = ( 1 - À ) L + LP, À G [0, 1[ with L = &L^(y, y) andP = y II B II 2 + f II B II f II B'B II f with B the set of bit sizes b of each element, B the average of the vector B and e of the hyperparameters, V being suitable for encouraging the reduction of the bit sizes b, and? allowing the polarization of said bit sizes to be encouraged;
[0028] - said decomposition is a decomposition into singular values, said threshold of predetermined bit size is equal to one, and the predetermined rank threshold is equal to 2222 with NXM the size of the tensor product of a tensor, having at most two dimensions or folded into two dimensions, of said at least one artificial neural network before said decomposition.
[0029] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement such a method for optimizing at least one artificial neural network as defined above.
[0030] The invention also relates to an electronic device for optimizing at least one artificial neural network during the training phase of said at least one artificial neural network, said neural network comprising an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer, said device comprising:
[0031] - a decomposition module configured to implement a decomposition tensor of said at least one artificial neural network, and for implementing training of said at least one decomposed artificial neural network;
[0032] - a quantization module configured to implement a quantization of at at least one element resulting from said decomposition using a bit size, said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function;
[0033] - a pruning module configured to prune said at least one neural network artificial provided by the quantization module by removing at least one element associated with a bit size less than a predetermined bit size threshold;
[0034] - a recomposition module configured to recompose at least one layer of said at least one pruned artificial neural network when after said pruning the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer and configured to train said at least one recomposed artificial neural network;
[0035] - an obtaining module configured to obtain said at least one neural network optimized by removing any previously applied quantization.
[0036] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:
[0037] - [Fig.l] [Fig.l] is a diagram illustrating a device for optimizing at least one artificial neural network according to one embodiment of the invention;
[0038] - [Fig.2] [Fig.2] is a flowchart of a method for optimizing at least one artificial neural network according to one embodiment of the invention;
[0039] - [Fig.3] [Fig.3] illustrates the application of the method of [Fig.2] to an example of artificial neural network.
[0040] Generally speaking, a neural network comprises an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer.
[0041] More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer, or from the input variables for the first layer.
[0042] Alternatively, more complex neural network structures can be envisaged with a layer that can be connected to a layer further away than the immediately preceding layer.
[0043] Each neuron is also associated with an operation, i.e. a type of processing, to be carried out by said neuron within the corresponding processing layer.
[0044] Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a connection between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.
[0045] Each neuron is capable of performing a weighted sum of the value(s) received from the neurons of the previous layer, each value then being multiplied by the respective synaptic weight of each synapse, or link, between said neuron and the neurons of the previous layer, then applying an activation function, typically a non-linear function, to said weighted sum, and delivering at the output of said neuron, in particular to the neurons of the following layer which are connected to it, the value resulting from the application of the activation function. The activation function makes it possible to introduce a non-linearity into the processing carried out by each neuron. The sigmoid function, the hyperbolic tangent function, the Heaviside function are examples of activation functions.
[0046] As an optional addition, each neuron is also capable of applying, in addition, an additive term, also called bias, to the input of the activation function, and the value delivered at the output of said neuron is then the result of the activation function applied to the sum of the bias value and the weighted sum.
[0047] A convolutional neural network is also sometimes referred to as a neural network convolutional or by the acronym CNN which refers to the English name of “Convo-lutional Neural Networks”.
[0048] In a convolutional neural network, each neuron in the same layer has exactly the same connection pattern as its neighboring neurons, but at different input positions. The connection pattern is called a convolution kernel, or more often a "kernel" in reference to the corresponding English term, or even a filter, since the convolution acts as a filter.
[0049] A convolutional network is capable of comprising several patterns or “kernels” per layer.
[0050] A fully connected layer of neurons is a layer in which the neurons of said layer are each connected to all the neurons of the previous layer.
[0051] Such a type of layer is more often referred to as “fully connected” and sometimes referred to as a “dense layer”.
[0052] [Fig.l] is a schematic representation of an embodiment of an electronic device 10 for optimizing at least one artificial neural network during the training phase of said at least one artificial neural network according to the present invention.
[0053] More precisely, such an electronic device 10 firstly comprises a decomposition module 12 configured to implement a tensor decomposition of said at least one artificial neural network and to implement training of said decomposed artificial neural network.
[0054] Such a decomposition module 12 is capable of providing a low-rank approximation via said decomposition, this low-rank approximation being advantageously considered, specifically according to the present invention, as a structured pruning, itself considered as a particular case of a low-granularity linear quantization.
[0055] Any tensor decomposition, suitable for compacting the energy of the different structures into a few significant elements which serve as a basis for approximately regenerating these elements, is suitable for use according to the present invention.
[0056] According to an optional and preferred embodiment, said decomposition is a singular value decomposition.
[0057] As an alternative, other decomposition variants are suitable for use according to the present invention such as: decomposition based on a discrete cosine transform (from the English Discrete Cosine Transform), or a discrete wavelet transform DWT (from the English Discrete Wavelet Transform), a canonical polyadic decomposition CP (from the English Canonical Polyadic Decom- position), Tucker decomposition, LU decomposition, etc.
[0058] Furthermore, the electronic device 10 for optimizing at least one artificial neural network comprises a quantization module 14 configured to implement a quantization of at least one element resulting from said decomposition, implemented by the decomposition module 12, using a bit size (from the English bit width), said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function.
[0059] As an optional addition, each bit size applied by the quantization module 14 is a decimal number, which allows for progressive convergence without abrupt discontinuity, particularly during forward / backward pass(es).
[0060] Such a quantization module 14 is in particular suitable for implementing an application (i.e. a quantization function) Qh which allows an association of a set of antecedents xeE (possibly infinite) to a finite set of discrete elements xq e F whose size is an increasing function of the width (i.e. size) of bit b. Such a quantization is suitable for being expressed in a general manner by means of the following expression: [ ° 061] Q h ( x )=Xi, ifre[A f , HAS z+1} avec AgR
[0062] As an optional variant, said quantification is affine according to the following formula: with je R, ae R+- and such that:
[0063] . x ~ ax clampl round ( ™ ), + aVeC e avec clamp a clipping function between two predetermined limits r / è) and Thi.bb corresponding to saturation thresholds, and round a function returning a rounded value.
[0064] More precisely, a is a quantization scale factor, P, by default zero, is a quantization alignment factor, the bit size b corresponds to the bit width of the structure considered, for example the neuron.
[0065] As an optional addition, the two saturation thresholds corresponding to the limits of the clipping function are expressed in the following form:
[0066] = -2^+l^Th(b) = 2W-1-
[0067] With regard to the cost function, according to an optional complement, the predetermined cost function is configured to encourage said at least one artificial neural network, during its training during said quantization, to reduce said bit size b. In other words, according to this optional complement, it is proposed to regularize the initial training cost function of the network by a term representative of its complexity. To do this, different cost functions are specific to be used according to the predetermined target objective corresponding for example to the optimization of the number of OPS operations per inference, the number of BOPS binary operations per inference, the memory footprint of the network, the L1 (denoted || || ^hereinafter) or L2 (denoted || || ^hereinafter) standards, a predetermined polarization term, the average or the sum of the bit widths, or the optimization of any possible combination of these different terms.
[0068] As an optional addition, the predetermined cost function is advantageously a regularized cost function (i.e. a regularization function) expressed in the following form:
[0069] LOSS = (1-À)L + ÀP, [0, 1[
[0070] with £ _ the original cost function that penalizes the model of neural network in the case of "bad" y, to which is added for regularization the penalty factor P = y || B || +1 || B || - || BB || , with B the set of bit sizes b of each element, B the average of the vector B and e R+2 of the hyperparameters, T being suitable for encouraging the reduction of the bit sizes b, and f allowing to encourage the polarization of said bit sizes. 2 allows to weight the regularization, the higher its value the more the neural network is compressed and vice versa. In other words, the compression rate is an increasing function with respect to 2. According to this option, as a bonus, the weighting term 2 is also bounded according to the expression of the cost function which makes it easier to automatically determine the ideal value for a given operating point.
[0071] In other words, the regularized cost function above uses a penalty factor P = yy B || -I-1 || B || - || B - B || mixing the L2 norm, with the term y || B || where is suitable for encouraging the reduction of bit sizes b (i.e. their amplitude), and the polarization term t || B || - || BB || , t allowing to encourage the polarization of bit sizes, noting that by "polarization" we mean the discrimination of useless neurons / structures whose bit size is pushed towards zero during said quantization with optimization. In other words, the polarization makes it possible to encourage the neural network during its training to separate itself from the useless neurons / structures corresponding to the term t || B || (the higher t is, the more the sparsity rate increases and vice versa) from those which are necessary for the proper functioning of the neural network corresponding to the term || B - B || .It is worth noting that the polarization term t || B || - || B - B || was introduced by T. Zhuang in the paper entitled “Neuron-level Structured Pruning using Polarization Regularizer . » relating only to pruning and not as proposed according to the present invention within the framework of a method cleverly combining by unification the three quasi-orthogonal solutions, namely the low rank approximation by decomposition, quantization and pruning as indicated previously.
[0072] Such a regularized cost function advantageously makes it easier to achieve convergence, in principle further limits the risk of local minimum and the number of hyperparameters to be determined, namely three corresponding to 2, t and F.
[0073] In other words, the three parameters a, [3, and b mentioned above are suitable for being learned during the training implemented within the quantization module, but only the parameter b is taken into account in the regularization function, called LOS S above.
[0074] It should be noted that the quantification module 14 uses an optimizer of said cost function, corresponding in particular optionally to a regularization function to determine the quantification parameters by structure.
[0075] To do this, according to a first option, the bit size is learned by said optimizer using a gradient descent algorithm, in particular a stochastic gradient descent SGD (from the English Stochastic Gradient Descent) or a variant thereof, known when implementing deep learning, in particular for other approaches adapted for small networks for example, such as an optimizer using an Adam optimization algorithm (from the English Adaptive Moment Estimation), a root mean square propagation algorithm RMSprop (from the English Root Mean Square propagation), simulated annealing, optimization by ant colony, optimization by particle swarms, genetic algorithms, etc.
[0076] As illustrated by [Fig. 1], the electronic device 10 for optimizing at least one artificial neural network also comprises a pruning module 16 configured to prune said at least one artificial neural network resulting from said quantification, implemented by the quantification module 14, by removing at least one element associated with a bit size less than a predetermined bit size threshold as illustrated below in relation to [Fig. 3].
[0077] As indicated previously, according to the present invention, the pruning implemented by the pruning module 16 is considered as a particular case of low-granularity quantization.
[0078] Furthermore, as previously indicated, the low-rank approximation by decomposition, implemented via the decomposition module 12, the quantization, implemented via the quantization module 14, and the pruning, implemented via the pruning module 16, are unified under the single prism of the low-granularity quantization which will impact the aforementioned cost function and consequently define the com- pressure associated with the artificial neural network.
[0079] In other words, the three approaches to optimizing a neural network, namely low-rank approximation, quantization and pruning are jointly optimized through the unique prism of low-granularity quantization during the training of the neural network considered to be optimized.
[0080] In particular, the optional term polarization / discrimination t || B || - || B - B || s of the aforementioned optional penalty factor P implemented by the quantification module 14 advantageously makes it possible to control the intensity of the subsequent pruning implemented by said pruning module 16.
[0081] As an optional addition, said predetermined bit size threshold implemented by said pruning module 16 is equal to one.
[0082] In the example of [Fig.l], the electronic device 10 for optimizing at least one artificial neural network also comprises, reciprocally to the decomposition module 12, a recomposition module 18 configured to recompose at least one layer of said at least one pruned artificial neural network when, after said pruning, implemented by the pruning module 16, the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer, and configured to train said at least one recomposed artificial neural network.
[0083] Indeed, according to the decomposition, implemented by the aforementioned decomposition module 12, the tensor product of each layer is divided into two stages:
[0084] According to an optional example where each tensor has at most two dimensions, for a matrix A of size NxM, which therefore requires NxM operations, it is possible to write A = A / x Ar with A / of size NxK and Ar of size KxM for a total number of operations of K(N + M) and K the rank of the decomposition.
[0085] If the rank K is sufficiently low, that is to say in the example if then the N+M decomposition has reduced the number of operations required. In the case where the number of post-decomposition operations is greater than pre-decomposition, decomposition is not interesting, and the recomposition module 18 is configured to recompose, calculating A; x Ar to restore the initial complexity, which ensures that the solution is not degraded.
[0086] Thus, optionally (i.e. according to the aforementioned optional example where each tensor has a maximum of two dimensions), the predetermined rank threshold used by the recomposition module 18 is equal to 2222 with N x. M the size of the tensor product of said at least one artificial neural network before said decomposition.
[0087] This principle can be generalized to any form of tensor of at least two dimensions, by passing through an intermediate representation in two dimensions 2D. In other words, in the case of a tensor of more than two dimensions, an inter transformation mediate consists of transforming a tensor of more than two dimensions by folding it into two dimensions. In this case, the predetermined rank threshold used by the recomposition module 18 is equal to with F and G the dimensions of the tensors transformed or folded into 2D.
[0088] In the example of [Fig.l], the electronic device 10 for optimizing at least one artificial neural network also comprises an obtaining module 20 configured to obtain said at least one optimized neural network by removing any previously applied quantization.
[0089] As an optional addition, the electronic device 10 for optimizing at least one artificial neural network also comprises a standardization module 22 configured to standardize each layer of said at least one retrained recomposed artificial neural network by determining a single bit size associated with each layer and to adjust the weights of said at least one artificial neural network standardized by retraining (i.e. finetuning). More precisely, said optional standardization module 22 is capable of intervening between said aforementioned recomposition module 18 and obtaining module 20.
[0090] In the example of [Fig.l], the electronic device 10 for optimizing at least one artificial neural network comprises a processing unit 24 formed for example by a memory 26 and a processor 28 associated with the memory 26, and the electronic device 10 for optimizing at least one artificial neural network is at least partly implemented in the form of software, or a software brick, executable by the processor, in particular the decomposition module 12, the quantization module 14, the pruning module 16, the recomposition module 18, the obtaining module 20 and optionally the standardization module 22. The memory 26 of the electronic device 10 for optimizing at least one artificial neural network is then capable of storing such software or software bricks, and the processor 28 is then capable of executing them.
[0091] When a part of the electronic device 10 for optimizing at least one artificial neural network according to the present invention is produced in the form of one or more software programs, i.e. in the form of a computer program, this part is furthermore capable of being recorded on a medium, not shown, readable by a computer. The computer-readable medium is, for example, a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. By way of example, the readable medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example EPROM, EEPROM, FLASH, NVRAM), a magnetic card or even an optical card. A program is then stored on the readable medium. computer containing software instructions.
[0092] An example of operation of the electronic device 10 for optimizing at least one artificial neural network according to the present invention is described below in relation to [Fig. 2] as illustrated previously in relation to [Fig. 1].
[0093] More precisely, the method 30 for optimizing at least one artificial neural network illustrated by [Fig.2] is implemented during the training phase of said at least one artificial neural network.
[0094] More precisely, the method 30 for optimizing at least one neural network firstly comprises a first step 32 of tensor decomposition D of said at least one artificial neural network and of training said at least one artificial neural network thus decomposed, said first step 32 being implemented by the decomposition module 12 mentioned above in relation to [Fig.l].
[0095] Optionally, said decomposition (32) is a singular value decomposition.
[0096] Then, the method 30 comprises a step 34 of quantification Q of at least one element resulting from said decomposition using a bit size, said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function. The quantification step 34 is implemented by the aforementioned quantification module 14 in relation to [Fig.l].
[0097] The quantification options previously cited in relation to the quantification module 14 of [Fig.l] are also suitable for being implemented during said method 30, namely independently or in combination as:
[0098] - each bit size applied during the quantization step is a number decimal;
[0099] - said quantification is affine according to the formula: with jg R, # ]^. and such that Qh:x•-»Xq = ax clamp^roimd(~~ ), + p' eR withclamp a clipping function between two predetermined limits T {b) and T}$), and round a function returning a rounded value, with optionally: [01°°] ^) = -2^ + 1^^)=2^
[0101] - the predetermined cost function is configured to incentivize said at least one artificial neural network, during its training during said quantization, to reduce said bit size, with optionally a predetermined cost function corresponding to a regularized cost function expressed in the following form: ( 1-2)L + 2P, 2 e [0, l[with£_ and P — y || B || + t || B || - || BB || , with B the set of bit sizes b of each element, B the mean of the vector B and y G r+2 of the hyperparameters, F being suitable for encouraging the reduction of bit sizes b, and enabling the polarization of said bit sizes to be encouraged.
[0102] Then, the method 30 comprises a step 36 of pruning E of said at least one artificial neural network resulting from said quantification 34 by removing at least one element associated with a bit size less than a predetermined bit size threshold.
[0103] Optionally, said predetermined bit size threshold is equal to one.
[0104] Then, the method 30 comprises a step 38 of recomposing at least one layer of said at least one pruned artificial neural network when after said pruning 36 the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer and training of said at least one recomposed artificial neural network.
[0105] Optionally, the predetermined rank threshold is equal to with NXM the size of the tensor product of a tensor, having at most two dimensions or folded into two dimensions as explained previously in the case where the tensor has more than two dimensions, of said at least one artificial neural network before said decomposition 32.
[0106] Optionally, as illustrated by the example of [Fig.2], the method 30 comprises, following said recomposition step 38 R, a step 40 of standardization U of each layer of said at least one recomposed artificial neural network retrained by determining a single bit size associated with each layer and adjusting the weights of said at least one artificial neural network standardized by retraining (from the English finetuning). More precisely, the single bit size retained during this standardization 40 corresponds, for each layer, to a bit size rounded to a fixed integer value.
[0107] The method 30 for optimizing at least one artificial neural network further comprises a step 42 of obtaining O of said at least one optimized neural network by removing any previously applied quantification. This step 42 of obtaining O is suitable for being implemented, in a manner not shown, directly after said recomposition step 38 R, or optionally as shown in [Fig.2] after the aforementioned optional step 40.
[0108] As indicated previously, the aforementioned order of combination of steps 32, 34, 36, 38, optionally 40, and step 42 of the method 30 according to the present invention is essential, because it makes it possible firstly to consider the low-rank approximation as structured pruning after the implementation of the first decomposition step, then to consider the structured pruning as a particular case of a low-granularity linear quantization.
[0109] [Fig.3] illustrates the application 44 of the method of [Fig.2] to an example of a network of artificial neurons 46 to be optimized comprising four layers, the first layer comprising two input neurons (neurons for a linear layer, filters for a convolutional layer), the second layer comprising four neurons, the third layer comprising three neurons and the fourth layer comprising two output neurons, said artificial neural network 46 to be optimized being received as input to said method.
[0110] The first decomposition step 32 of [Fig.2] is firstly illustrated by the arrow 48, the white color of which indicates that this step does not include training.
[0111] The decomposition result 50, in particular a decomposition into singular values, is then quantified according to the arrow 52, the black color of which is representative of the fact that said quantification step further comprises training said neural network until optimal quantification is obtained.
[0112] The result 54 of said quantization represents each element of the previous decomposition result 50 which is associated with a bit size, represented in [Fig.3] by means of a bit size scale, here illustrated via a texture scale 56, a zero bit size being represented by the black texture 560, a bit size equal to one being represented by the texture 562, a bit size equal to two being represented by the texture 564, a bit size equal to three being represented by the texture 566, a bit size equal to four being represented by the white texture 568.
[0113] It should be noted that the example of [Fig.3] based on bit widths between zero and four bits is only illustrative, and that the present invention is suitable for application to other bit widths.
[0114] It should be noted that optionally and advantageously the bit size is a decimal number so that the set of decimal values of the bit size scale 56 with intermediate textures, not shown for the sake of simplicity, is suitable for being used in an illustrative manner to illustrate the result 54 of said quantization. According to another representation, an illustrative scale based on shades of color, in particular shades of gray, would be suitable for illustrating the quantization with a bit size corresponding to a decimal number, each change of decimal being associated with a distinct shade of color, in particular gray. Such a quantization with a bit size corresponding to a decimal number allows for progressive convergence without abrupt discontinuity, in particular during forward / backward pass(s).
[0115] The subsequent pruning step is then applied according to the arrow 58, the black color of which is representative of the fact that said pruning step further comprises training said neural network until optimal pruning 60 is obtained where the elements of the quantized neural network 54 whose bit size is less than one predetermined bit size threshold, in particular equal to one and represented by the color black, are all deleted, so that for example the pruning result 60 includes eleven neurons against eighteen in the quantization result 56 and also that of decomposition 50.
[0116] The subsequent recomposition step is then applied according to arrow 62, the white color of which indicates that this step does not include training. The second and third layers have thus been recomposed to arrive at the recomposition result 64 which now only includes eight neurons.
[0117] The optional standardization step is then applied according to the arrow 66, the black color of which is representative of the fact that said optional standardization step further comprises training said neural network to adjust (from the English finetuning) the weights of the model and obtain the result 68 illustrating in particular the fact that the bit size of the second layer is now common to the two neurons of the second layer whereas it was different in the previous result 64 of recomposition, the bit size rounded to the largest integer value having been retained during the optional standardization 66.
[0118] For the sake of simplicity, the result of the obtaining step is not represented but is easily deduced because it consists of removing any quantification (represented on the standardization result 68 by the shades of gray used to represent each neuron point).
[0119] Those skilled in the art will understand that the invention is not limited to the embodiments described, nor to the particular examples of the description, the embodiments and variants mentioned above being suitable for being combined with each other to generate new embodiments of the invention.
[0120] The present invention thus makes it possible to cleverly unify three quasi-orthogonal neural network compression solutions of the state of the art under the single prism of low-granularity quantification, which greatly simplifies the search for hyperparameters compared to the case where such unification is not implemented, the sequential juxtaposition of the three neural network compression solutions and the associated search for hyperparameters then being much more complex in comparison.
[0121] In other words, the present invention proposes to cleverly and jointly mix three quasi-orthogonal optimizations for greater theoretical maximum gains and a reduced risk of local optimum, and their unification under the unique prism of low-granularity quantization makes it possible to reduce the number of necessary hyperparameters.
Claims
Claims
1. Method (30) for optimizing at least one artificial neural network, said method being implemented during the training phase of said at least one artificial neural network, said neural network comprising an ordered succession of layers of neurons each of which takes its inputs from the outputs of the previous layer, said method being implemented by an electronic device for optimizing at least one artificial neural network, and comprising the following successive steps: - tensor decomposition (32) of said at least one artificial neural network and training of said at least one decomposed artificial neural network; - quantification (34) of at least one element resulting from said decomposition using a bit size, said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function;- pruning (36) said at least one artificial neural network resulting from said quantification by removing at least one element associated with a bit size less than a predetermined bit size threshold; - recomposing at least one layer of said at least one pruned artificial neural network when after said pruning the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer and training said at least one recomposed artificial neural network; - obtaining said at least one optimized neural network by removing any previously applied quantification.;
2. Method (30) according to claim 1, further comprising, between said recomposition step (38) and said obtaining step (42), a step of standardizing (40) each layer of said at least one recomposed artificial neural network retrained by determining a unique bit size associated with each layer and adjusting the weights of said at least one artificial neural network standardized by retraining.
3. The method (30) of claim 1 or 2, wherein each bit size applied in the quantizing step is a decimal number.
4. A method (30) according to any preceding claim, wherein said quantization is affine according to the formula: with xeR, « £ R+, and such that Qh: x xq = ax damp( round (^), Ti$)} + Z? R with clamp a clipping function between two predetermined limits T and T^b), and round a function returning a rounded value.
5. The method (30) of claim 4, wherein T / (b) = - 2^+ 1 ct r„(Z,)=2M-b
6. Method (30) according to any one of the preceding claims, in which the predetermined cost function is a regularized cost function expressed in the following form: LOSS = ( 1 - 2 )L + ÀP, 2 e [0, 1[ withL= iYZyUy, y) II B II 2 + / Il B II r II B~B II 1' with B the set of bit sizes b of each element, B the average of the vector B and ( ye r+2 of the hyperparameters, Y being suitable for incentivizing the reduction of the bit sizes b, and? making it possible to incentivize the polarization of said bit sizes.
7. A method (30) according to any preceding claim, wherein said decomposition (32) is a singular value decomposition, said predetermined bit size threshold is equal to one, and the predetermined rank threshold is equal to with N x M the size of the tensor product of a tensor, having at most two dimensions or folded into two dimensions, of said at least one artificial neural network before said decomposition (32).
8. Computer program comprising software instructions which when executed by a computer, implement, at least in part, optimization of at least one artificial neural network according to any one of the preceding claims.
9. Electronic device (10) for optimizing at least one artificial neural network during the training phase of said at least one artificial neural network, said neural network comprising an ordered succession of layers of neurons each of which takes its inputs from the outputs of the previous layer, said device being characterized in that it comprises: - a decomposition module (12) configured to implement a tensor decomposition of said at least one artificial neural network, and to implement training of said at least one decomposed artificial neural network; - a quantization module (14) configured to implement a quantization of at least one element resulting from said decomposition using a bit size, said bit size being learned by training said at least one artificial neural network by optimizing a predetermined cost function; - a pruning module (16) configured to prune said at least one artificial neural network provided by the quantization module by removing at least one element associated with a bit size less than a predetermined bit size threshold; - a recomposition module (18) configured to recompose at least one layer of said at least one pruned artificial neural network when, after said pruning, the rank of said layer is greater than a predetermined rank threshold associated with said at least one layer and configured to train said at least one recomposed artificial neural network; - an obtaining module (22) configured to obtain said at least one optimized neural network by removing any previously applied quantization.