Systems and methods for compiling trained artificial neural networks

Abstract

Embodiments of the present disclosure relate to systems and methods for compiling a trained artificial neural network. According to one aspect, a computer-implemented method compiles a first trained artificial neural network including at least one contiguous layer including a deep convolutional layer, a saturated rectified linear unit layer, and a two-dimensional convolutional layer. The method comprises: balancing between layers; replacing the saturation rectification linear unit layer with a self-adaptive channel trimming layer to obtain an artificial neural network with modified topology; performing tensor quantization on the layers of the artificial neural network with the modified topology; and compiling the quantized artificial neural network with the modified topology.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of French Patent Application No. 2310810, filed on October 10, 2023, which is hereby incorporated herein by reference. Technical Field

[0003] Embodiments relate to artificial neural networks. Background Art

[0004] Artificial neural networks typically include a series of neuron layers. Each layer takes as input data with weights applied, and delivers as output data after being processed by a function for activating the neurons of that layer. These output data (also called "activations") are transmitted to the next layer in the neural network.

[0005] Weights are the data of neurons, more specifically the parameters of neurons, which can be configured to obtain good data at the output of each layer. The weights of a layer are defined in a weight tensor. A weight tensor can have multiple channels.

[0006] The weights are adjusted during a general supervised learning phase, in particular by running the neural network using classified data from a reference database as input data. This learning phase enables a trained neural network to be obtained.

[0007] Once a neural network has been trained, it can be integrated into an integrated circuit such as a microcontroller.

[0008] In particular, artificial neural networks can be integrated into integrated circuits using integration software. For example, the integration software STM32Cube.AI and its extension X-CUBE-AI developed by STMicroelectronics are known.

[0009] In addition, some integrated circuits include artificial neural network processing circuits, also known as artificial neural network accelerators. This specific circuit is designed to improve the runtime performance of artificial neural networks. This circuit can be designed in particular according to a specific hardware architecture to optimize common operations of artificial neural networks, such as matrix multiplication, activation functions, and convolution operations. This architecture, for example, enables efficient parallel computing. In this way, this circuit is designed to maximize performance in terms of computing speed and energy efficiency. This significantly speeds up the operation of artificial neural networks.

[0010] Furthermore, artificial neural networks are often trained using floating point representations for their weights and activations. Using floating point representations often results in trained artificial neural networks with higher accuracy.

[0011] However, artificial neural network processing circuits are typically designed to run integer or fixed-point quantized neural networks. Artificial neural network processing circuits are typically not designed to run artificial neural networks using floating-point representations.

[0012] In particular, artificial neural networks can be quantized to speed up their operation and reduce memory requirements. Neural network quantization involves defining the format used to represent neural network data, such as weights and inputs and outputs of each layer of the neural network. Quantization enables, in particular, switching from floating point representation to fixed point number format. The layers of the neural network can be quantized to, for example, eight bits or binary. Quantization of neural networks can be achieved by integrated software.

[0013] The quantization may be performed during the learning phase or after the learning phase.

[0014] The acronym "QAT" (Quantization Aware Training) may be used to describe quantization during the learning phase. Quantization during the learning phase requires retraining the trained artificial neural network by simulating quantization. This quantization is complex, requires a lot of computation to perform the learning phase, and may also require tuning of hyperparameters. This quantization also has the disadvantage of requiring all of the training data. Nevertheless, quantization during the learning phase brings the performance of the quantized trained neural network closer to the performance of a trained neural network using floating point representations for its weights and activations.

[0015] The quantization performed after the learning phase can be referred to using the acronym "PTQ" (Post-Training Quantization). The quantization is performed using the trained neural network and requires little or no data. The quantization also requires minimal adjustments to the hyperparameters of the artificial neural network. The quantization does not require a new learning phase for the artificial neural network at a later time. This makes it easier to quantize the artificial neural network than if it were quantized during the learning phase.

[0016] Furthermore, quantization can be tensor quantization or channel quantization.

[0017] Tensor quantization uses a set of weight tensor quantization parameters. In other words, weights of the same tensor share the same scale and the same bias.

[0018] Channel quantization uses a separate set of parameters for each output channel of the weight tensor. In other words, weights of the same channel of the weight tensor share the same scale and the same bias, but the scale and bias may be different for different channels of the weight tensor. Channel quantization increases the granularity of quantization. This generally improves the accuracy of the quantized neural network compared to tensor quantization.

[0019] However, channel quantization is complex to implement on artificial neural network processing circuits. In particular, channel quantization may require scaling when using data that does not have the same scale. Therefore, when integrating artificial neural networks into integrated circuits such as microcontrollers using neural network processing circuits, tensor quantization is more often chosen.

[0020] The publication "A White Paper on Neural Network Quantization" by Markus Nagel et al., published on the website arXiv.org on June 15, 2021, describes a solution for improving the accuracy of tensor quantized artificial neural networks to bring it close to the accuracy of trained artificial neural networks using floating point representation. In particular, the publication describes a quantization method using cross-layer equalization and bias absorption.

[0021] However, the solutions proposed in this publication have proven to be insufficient in certain situations.

[0022] Therefore, there is a need to improve tensor quantization for artificial neural networks. Summary of the invention

[0023] According to one aspect, a method for compiling a first trained artificial neural network executed by a computer is provided, the first trained artificial neural network comprising at least one succession of layers, the at least one succession of layers comprising a deep convolution layer, then a saturated rectified linear unit layer, and then a two-dimensional convolution layer, the method comprising: performing equalization between the deep convolution layer and the two-dimensional convolution layer, replacing the saturated rectified linear unit layer with an adaptive channel pruning layer to obtain an artificial neural network with a modified topology, performing tensor quantization on the layers of the artificial neural network with the modified topology, and compiling the quantized artificial neural network with the modified topology to generate a computer program, the computer program comprising instructions, which, when the program is executed by the computer, causes the computer to implement the quantized artificial neural network with the modified topology.

[0024] The adaptive channel pruning layer avoids the need to use a saturated rectified linear unit layer while preserving the original behavior of the artificial neural network.

[0025] The compilation method thus enables tensor quantization after training an artificial neural network, while limiting the accuracy loss of the quantized artificial neural network compared to an artificial neural network trained using floating point representation for its weights and activations.

[0026] The advantage of doing this tensor quantization after training the ANN is that it can be performed without all the data used to perform the training.

[0027] The advantage of this tensor quantization is that it is easy to implement compared to quantization during the learning phase.

[0028] According to an advantageous embodiment, the adaptive channel pruning layer is adapted according to a saturation value defined for the saturated rectified linear unit layer.

[0029] The adaptive channel pruning layer is preferably designed to prune each channel of the activation tensor generated by the deep convolutional layer between a minimum value equal to 0 and a maximum value calculated by applying the following formula:

[0030] max_val = X × s i -1 , where max_val is the maximum value, X is the saturation value defined by the Saturated Rectified Linear Unit layer, and where r i (1) corresponds to the range of values ​​of channel i of the weight tensor of the depthwise convolutional layer, and r i (2) Corresponds to the range of values ​​for channel i of the weight tensor of the 2D convolutional layer.

[0031] In an advantageous embodiment, the method further comprises: performing bias absorption between the depthwise convolutional layer and the two-dimensional convolutional layer.

[0032] Advantageously, the method further comprises:

[0033] - Obtaining an initially trained artificial neural network, the initially trained neural network comprising at least one succession of layers, the at least one succession of layers comprising a deep convolutional layer, then a batch normalization layer, then a saturated rectified linear unit layer, then a two-dimensional convolutional layer, then possibly a second batch normalization layer, and fusing the first batch normalization layer of the initially trained artificial neural network with the deep convolutional layer, and possibly fusing the second batch normalization layer with the two-dimensional convolutional layer, thereby obtaining a first trained artificial neural network.

[0034] According to another aspect, a compilation computer program product is provided. The compilation computer program product comprises instructions. When the program is executed by a computer, the instructions enable the computer to implement the compilation method as described above.

[0035] According to another aspect, a quantized and trained artificial neural network is proposed, the neural network comprising at least one successive layer comprising a deep convolutional layer, then an adaptive channel pruning layer, then a two-dimensional convolutional layer.

[0036] According to another aspect, a computer system is proposed, the computer system comprising a memory storing a compiled computer program product as described above, and a main processor designed to run the compiled computer program.

[0037] According to another aspect, a computer-readable data medium is proposed, on which a compiled computer program product as described above is stored.

[0038] According to another aspect, a computer program product is proposed, which comprises instructions for directing the computer to implement a quantized artificial neural network with a modified topology obtained according to the compilation method as described above when the program is executed by the computer.

[0039] According to another aspect, a microcontroller is proposed, comprising a memory storing a computer program product as described above for running a quantized artificial neural network with a modified topology and a main processor designed to run the computer program.

[0040] In an advantageous embodiment, the microcontroller further comprises a neural network processing circuit designed to run a quantized artificial neural network having a modified topology.

[0041] According to another aspect, a computer-readable data medium is proposed, on which a computer program product as described above is stored in order to run a quantized artificial neural network with a modified topology. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other advantages and features of the invention will become apparent on reading the detailed description of embodiments (which are in no way limiting) and the detailed description of the accompanying drawings, in which:

[0043] Figure 1 An embodiment of a computer system is illustrated;

[0044] Figure 2 An embodiment of a method for compiling an artificial neural network is illustrated;

[0045] Figure 3 The compilation method is illustrated;

[0046] Figure 4 An artificial neural network with a series of layers is illustrated;

[0047] Figure 5 illustrates an artificial neural network having a series of layers; and

[0048] Figure 6An embodiment of a microcontroller configured to run a quantized artificial neural network is illustrated. DETAILED DESCRIPTION

[0049] Figure 1 An embodiment of a computer system SYS is shown. The computer system SYS comprises a main processor CPU1 and a memory MEM1. The memory MEM1 stores a compiled computer program PRG1, which comprises instructions which, when the program is run by the main processor CPU1, cause the main processor CPU1 to implement a method for compiling an artificial neural network, as described below with reference to Figure 2 as described.

[0050] The compiled computer program PRG1 may be included in the integration software. The integration software is designed to integrate the quantized neural network into an integrated circuit.

[0051] Figure 2 An embodiment of a method for compiling an artificial neural network is shown. The method can be implemented by running a compiling computer program PRG1.

[0052] The method comprises obtaining 10 a trained artificial neural network TNN. The trained artificial neural network TNN may be provided as input to the integration software.

[0053] The trained artificial neural network TNN comprises a series of neuron layers. Each layer takes as input the data to which weights are applied, and delivers as output the data after being processed by the function for activating the neurons of that layer. These output data (also referred to as "activations") are transmitted to the next layer in the neural network. In particular, the trained artificial neural network TNN then comprises an input layer LAY_I and an output layer LAY_O, the input layer LAY_I being designed to receive input data from the artificial neural network TNN, and the output layer LAY_O being designed to generate output data from the artificial neural network TNN.

[0054] The weights of a layer are defined in a weight tensor. A weight tensor can have multiple channels. The weights are the data of a neuron, more specifically the parameters of a neuron, which can be configured to obtain good data at the output of each layer.

[0055] The weights are adjusted in a generally supervised learning phase, in particular by running the neural network using classified data from a reference database as input data. This learning phase enables a trained neural network to be obtained.

[0056] The training aims to define the neural network with the best performance. More specifically, during the training of the neural network, the weights of the neural network are adjusted and then the performance of the modified network is evaluated. If the performance of the neural network is satisfactory, the training of the network is terminated.

[0057] A trained neural network may use floating point representations for the weights as well as the data (activations) generated by each neural network layer.

[0058] The trained neural network has a format known to those skilled in the art. For example, the trained neural network can be obtained from artificial neural network design software such as Keras, Tensorflow, and ONNX (abbreviation for "OpenNeural Network Exchange").

[0059] like Figure 2 As shown in , the artificial neural network comprises at least one succession of layers SUCC_1, called the succession of layers of interest, comprising a deep convolutional layer D_CONV, then a first batch of normalization layers B_NORM1, then a saturated rectified linear unit layer RELUX, then a two-dimensional convolutional layer CONV_2D, then possibly a second batch of normalization layers B_NORM2. Figure 2 In the figure, only one interesting layer of the chain SUCC_1 is shown. However, the artificial neural network may include multiple interesting layers of the chain, such as Figure 2 as shown in .

[0060] The depth convolution layer D_CONV is designed to apply a convolution kernel to each input channel of the depth convolution layer. Therefore, a convolution kernel is defined for each input channel of the depth convolution layer.

[0061] The first batch normalization layer B_NORM1 is designed to normalize the tensor activations it receives as input. In particular, the batch normalization layer is designed to calculate the mean and standard deviation of the activations, and then normalize the activations by subtracting the activations from the calculated mean and dividing the activations by the calculated standard deviation. The first batch normalization layers are used to reduce the drift effect of the distribution of activations during the learning phase of the artificial neural network. The first batch normalization layers are also used to accelerate convergence during training. This faster convergence means that the artificial neural network can be trained faster. This also helps to reduce the phenomenon of overfitting. Overfitting corresponds to adjustments to the artificial neural network during training that are too closely linked to the training data (e.g., training images). When the artificial neural network is run with input data (e.g., new images) that is different from the training data, this overfitting can lead to a reduction in the accuracy of the artificial neural network. In other words, overfitting means that the artificial neural network loses its ability to generalize to data that is different from the data used for training.

[0062] The saturated rectified linear unit layer RELUX helps introduce nonlinearity into artificial neural networks. This nonlinearity enables artificial neural networks to learn complex functions.

[0063] In particular, the Saturated Rectified Linear Unit layer is designed to perform the following function:

[0064] ReLUX(z)=min(max(0,z),X), where z corresponds to the output of the first normalization layer on channel i, ReLUX(z) corresponds to the activation of the output tensor of the first normalization layer on channel i, and X corresponds to the saturation value.

[0065] In other words, a saturated rectified linear unit layer RELUX is used to generate an output tensor in which the value of the activation of each channel of the output tensor corresponds to a minimum value between a defined saturation value X and a maximum value between 0 and the corresponding value of the activation of the associated channel of the input tensor.

[0066] For example, the saturated rectified linear unit layer at 6 (“ReLU6”) is designed to perform the following function:

[0067] ReLU6(z)=min(max(0,z),6), where z corresponds to the output of the first normalization layer on channel i, ReLU6(z) corresponds to the activation of the output tensor of the first normalization layer on channel i, and 6 corresponds to the saturation value.

[0068] The 2D convolutional layer CONV_2D is designed to apply the same convolution kernel to all channels of the activation tensor that it receives as input.

[0069] Similar to the first batch normalization layer B_NORM1, the second batch normalization layer B_NORM2 is designed to normalize the tensor activations received as input. In particular, the batch normalization layer is designed to calculate the mean and standard deviation of the activations and then normalize the activations by subtracting the activations from the calculated mean and dividing the activations by the calculated standard deviation. The second batch normalization layer is used to reduce the drift effect of the distribution of activations during the learning phase of the artificial neural network. The second batch normalization layer is also used to speed up convergence during training. This faster convergence means that the artificial neural network can be trained faster. This also reduces the phenomenon of overfitting.

[0070] The compilation method comprises detecting 11 the layers of interest of at least one succession SUCC_1. In this step, the main processor CPU1 detects the layers of interest of each succession SUCC_1, the succession SUCC_1 thus comprising a deep convolutional layer D_CONV, then a first batch of normalization layers B_NORM1, then a saturated rectified linear unit RELUX, then a two-dimensional convolutional layer CONV_2D, then possibly a second batch of normalization layers B_NORM2. This detection is performed based on information associated with each layer of the trained artificial neural network.

[0071] Then, the main processor CPU1 executes steps 12 to 15 described below for each detected layer of interest of a series of SUCC_1, such as Figure 3 as shown in .

[0072] In particular, the method comprises a step 12 for fusing the batch normalization layers B_NORM1, B_NORM2 with their preceding layers D_CONV, CONV_2D. This step 12 is performed by the main processor CPU1. This fusion can be called "batch normalization folding". The fusion removes the operation of each batch normalization layer in the artificial neural network, thereby obtaining a series of SUCC_2 layers, such as Figure 4 In particular, the operations of each batch normalization layer are integrated into the layer before the batch normalization layer. This reduces the number of computations required to perform each batch normalization layer.

[0073] In particular, the weights and biases associated with the layers preceding each batch normalization layer are recomputed in order to integrate the operation of the batch normalization layer. In particular, the weights of the layers preceding the batch normalization layer are computed using the following function:

[0074] where γ k is a hyperparameter whose value is determined by training, W k is the value of the previous weight, σ is the variance calculated during training, and ε is a predefined constant.

[0075] Use the following function to calculate the bias of the layer before the batch normalization layer: where b k is the initial value of the bias, β k is the first hyperparameter of the batch normalization layer, whose value is defined by training, γ k is the second parameter of the batch normalization layer, μ k is the mean value calculated during training, σ k is the variance calculated during training, and ε is a predefined constant.

[0076] The method then comprises a step 13 of cross-layer equalization. This step 13 of equalization is performed by the main processor CPU 1. The step 13 of cross-layer equalization corresponds to the method described in the publication “A White Paper on Neural Network Quantization” of June 15, 2021 by Markus Nagel et al.

[0077] Cross-layer equalization is designed to adapt the weights and biases of two consecutive layers of an artificial neural network, for example, between a deep convolutional layer D_CONV and a two-dimensional convolutional layer CONV_2D, separated by a saturated rectified linear unit layer RELUX.

[0078] In particular, for two consecutive layers of an artificial neural network, the output activation tensor of the first layer can be defined by the following expression.

[0079] h=f(W (1) x+b (1) ), where f is the activation function of the first layer, W (1) is the initial weight tensor of the first layer, x corresponds to the activation tensor that is the input to the first layer, and b (1) is the bias tensor for the first layer.

[0080] The output activation tensor of the second layer can be defined by the following expression.

[0081] y=f(W (2) h+b (2) ), where f is the activation function of the second layer, W (2) is the initial weight tensor of the second layer, h corresponds to the activation tensor that is the input to the second layer, and b (2) is the bias tensor of the second layer.

[0082] By applying cross-layer equalization, the weight tensors of two consecutive layers can be redefined so that in The new weight tensor corresponding to the second layer, The new weight tensor corresponding to the first layer, The new bias tensor corresponding to the first layer, corresponds to the adaptive pruning function described below, and where S = diag(s) is a diagonal matrix whose values ​​S ii Corresponding to the scale factor s i .

[0083] In particular, the scaling factor s i is defined to equalize the range of values ​​between different successive layers. More specifically, the scaling factor s i Obtained by the following formula:

[0084] where r i (1) corresponds to the range of values ​​of channel i of the weight tensor of the depthwise convolutional layer, and r i (2) Corresponds to the range of values ​​for channel i of the weight tensor of the 2D convolutional layer.

[0085] Such a scaling factor makes it possible to improve the robustness of the quantization by reducing the quantization noise.

[0086] Cross-layer equalization helps avoid the problem of amplitude differences between elements of the same tensor. This problem often occurs when deep convolutional layers are used. Therefore, cross-layer equalization helps reduce the accuracy loss that may be caused by tensor quantization.

[0087] The method then comprises a step 14 of bias absorption. This step 14 of bias absorption is performed by the main processor CPU 1. This step 14 of bias absorption corresponds to the method described in the publication “A White Paper on Neural Network Quantization” of June 15, 2021 by Markus Nagel et al.

[0088] Bias absorption is designed to modify parameters using the following formula in order to absorb the bias: in

[0089] where c i =max(0,β i -3γ i ), β i and γ i are two hyperparameters of the batch normalization layer BNORM1 for channel i. The value “3” can also be adjusted by extracting activation statistics and using simulations.

[0090] Furthermore, if the saturated rectified linear unit layers frequently lead to saturation, cross-layer equalization may not be sufficient to avoid the accuracy loss caused by tensor quantization.

[0091] To solve this problem, the method then comprises a step 15 for replacing the saturated rectified linear unit layer. This step 15 is performed by the main processor CPU1. In this step, the saturated rectified linear unit layer is replaced by an adaptive channel pruning layer CPL_L, thereby obtaining a series of SUCC_3 layers, such as Figure 5 As shown in . Therefore, this step leads to a modification of the topology of the artificial neural network.

[0092] The adaptive channel pruning layer CPL_L is designed to prune the output tensor of the depthwise convolutional layer.

[0093] In particular, each channel of the tensor generated by the deep convolutional layer is pruned independently between a minimum and a maximum value. The minimum value is constrained to 0. The maximum value is calculated as follows: max_val = X × s i -1 , where max_val is the maximum value, X is the saturation value defined by the replaced saturated rectified linear unit layer, and s i is the value calculated for cross-layer equalization of channel i.

[0094] The adaptive channel pruning layer avoids the need to use a saturated rectified linear unit layer while preserving the original behavior of the artificial neural network.

[0095] The method then comprises a step 16 of tensor quantization of the layers of the artificial neural network. This step 16 is performed by the main processor CPU1.

[0096] In this way, this method makes it possible to obtain a quantized and trained artificial neural network QNN having a level of accuracy close to that of an artificial neural network trained using floating point representation for its weights and its activations.

[0097] Then, the method comprises a step 17 for compiling the quantized and trained artificial neural network QNN. This step 17 is performed by the main processor CPU1. This step 17 enables the generation of a computer program PRG2, which contains instructions that, when the program is run by a computer, direct the computer to implement the obtained quantized artificial neural network QNN. In particular, the computer program QNN may contain instructions so that the quantized artificial neural network can be run by the artificial neural network processing circuit. Therefore, this computer program PRG2 is stored in the memory MEM1 of the computer system SYS.

[0098] Then, the computer program PRG2 can be integrated into an integrated circuit such as a microcontroller MCU. Then, the integrated circuit can implement the quantized artificial neural network QNN by executing the instructions of the computer program PRG2.

[0099] The above-described compilation method enables tensor quantization after training an artificial neural network, while limiting the accuracy loss of the quantized artificial neural network compared to an artificial neural network trained using floating point representation for its weights and its activations.

[0100] The advantage of doing this tensor quantization after training the artificial neural network is that it can be performed without all the data used to perform the training. A limited amount of data used for training may still be needed to calibrate the layer outputs and activations.

[0101] The advantage of this type of tensor quantization is that it is easier to implement compared to quantization during training.

[0102] Furthermore, the modified topology of the artificial neural network has a minimal impact on the running time of the artificial neural network and on the amount of data that has to be stored in the flash memory of the integrated circuit in which the artificial neural network is integrated.

[0103] Of course, the embodiments of the above-mentioned compilation method may be subjected to various variations and modifications, which will be apparent to those skilled in the art. In particular, the processor may perform steps 13 to 15 only for certain series of layers of interest. In particular, the computer program PRG1 may be designed to determine which series of layers of interest may include a saturated rectified linear unit layer, the saturation frequency of which during the operation of the trained neural network is greater than a predefined threshold. More specifically, the saturation may be evaluated by running the trained artificial neural network using input data that has been used during the learning phase of the artificial neural network. Then, the computer program PRG1 may be designed to perform steps 13 to 15 only for the layers of interest of the series SUCC_1 that may include a saturated rectified linear unit layer, the saturation frequency of which during the operation of the trained neural network is greater than a predefined threshold. The computer program PRG1 may also be designed to modify the saturated rectified linear unit layer of the layers of interest of other series SUCC_1 by a non-saturated rectified linear unit layer (i.e., "ReLU" type).

[0104] Figure 6 An embodiment of a microcontroller MCU is shown, and the microcontroller MCU is designed to operate through the above reference Figure 3 The quantized artificial neural network QNN is obtained by the described method.

[0105] The microcontroller comprises a main processor CPU2, a memory MEM2 and an artificial neural network processing circuit ACC. The memory MEM2 stores a computer program PRG2, which comprises instructions that, when the program is executed by the neural network processing circuit ACC, enable the neural network processing circuit ACC to implement a quantized artificial neural network QNN.

Claims

1. A computer-implemented method for compiling a first trained artificial neural network, the first trained artificial neural network comprising at least one succession of layers, the at least one succession of layers comprising a depthwise convolutional layer, then a saturated rectified linear unit layer, and then a two-dimensional convolutional layer, the method comprising: Using a processor, performing balancing between the depthwise convolution layer and the two-dimensional convolution layer; replacing the saturated rectified linear unit layer with an adaptive channel pruning layer using the processor to obtain an artificial neural network with a modified topology; performing tensor quantization on the layer of the artificial neural network having the modified topology using the processor to obtain a quantized artificial neural network having the modified topology; as well as The quantized artificial neural network with the modified topology is compiled by the processor to generate a computer program, wherein the computer program includes instructions that, when the computer program is executed by a computer, cause the computer to implement the quantized artificial neural network with the modified topology.

2. The method of claim 1, wherein the adaptive channel pruning layer is adapted according to a saturation value defined for the saturated rectified linear unit layer.

3. The method of claim 2, wherein the adaptive channel pruning layer is configured to prune each channel of an activation tensor generated by the deep convolutional layer between a minimum value and a maximum value, wherein the minimum value is equal to 0 and the maximum value is calculated by applying the following formula: where max_val is the maximum value, X is the saturation value defined by the Saturated Rectified Linear Unit layer, and where r i (1) corresponds to a first range of values ​​for channel i of the first weight tensor of the depthwise convolutional layer, and r i (2) A second range of values ​​corresponding to channel i of the second weight tensor of the two-dimensional convolutional layer.

4. The method according to claim 1, further comprising: A bias absorption is performed between the depthwise convolution layer and the two-dimensional convolution layer.

5. The method according to claim 1, further comprising: Obtaining an initially trained artificial neural network, the initially trained artificial neural network comprising at least one initial series of layers, the at least one initial series of layers comprising an initial depthwise convolutional layer, then a first batch of normalized layers, then an initial saturated rectified linear unit layer, and then an initial two-dimensional convolutional layer; as well as The first batch of normalization layers of the initially trained artificial neural network are fused with the initial depth convolutional layer to obtain a first trained artificial neural network.

6. The method of claim 5, wherein the at least one initial series of layers further comprises a second batch of normalization layers, the second batch of normalization layers following the initial two-dimensional convolutional layer, and the method further comprises: The second batch of normalization layers are fused with the initial two-dimensional convolutional layer to obtain a first trained artificial neural network.

7. A non-transitory computer readable medium storing computer instructions for compiling a first trained artificial neural network, the first trained artificial neural network comprising at least one series of layers, the at least one series of layers comprising a depthwise convolutional layer, then a saturated rectified linear unit layer, and then a two-dimensional convolutional layer, the computer instructions, when executed by a processor, causing the processor to perform the following steps: Performing balancing between the depth convolution layer and the two-dimensional convolution layer; replacing the saturated rectified linear unit layer with an adaptive channel pruning layer to obtain an artificial neural network with a modified topology; performing tensor quantization on the layer of the artificial neural network having the modified topology to obtain a quantized artificial neural network having the modified topology; and The quantized artificial neural network with the modified topology is compiled to generate a computer program, wherein the computer program includes compilation instructions, and when the computer program is executed by a computer, the compilation instructions enable the computer to implement the quantized artificial neural network with the modified topology.

8. The non-transitory computer readable medium of claim 7, wherein the adaptive channel pruning layer is adapted according to a saturation value defined for the saturated rectified linear unit layer.

9. The non-transitory computer-readable medium of claim 8, wherein the adaptive channel pruning layer is configured to prune each channel of an activation tensor generated by the deep convolutional layer between a minimum value and a maximum value, the minimum value being equal to 0, the maximum value being calculated by applying the following formula: where max_val is the maximum value, X is the saturation value defined by the Saturated Rectified Linear Unit layer, and where r i (1) corresponds to a first range of values ​​for channel i of the first weight tensor of the depthwise convolutional layer, and r i (2) A second range of values ​​corresponding to channel i of the second weight tensor of the two-dimensional convolutional layer.

10. The non-transitory computer readable medium of claim 7, comprising further computer instructions which, when executed by the processor, cause the processor to perform the step of bias absorption between the depthwise convolutional layer and the two-dimensional convolutional layer.

11. The non-transitory computer readable medium of claim 7, comprising further computer instructions which, when executed by the processor, cause the processor to perform the following steps: Obtaining an initially trained artificial neural network, the initially trained artificial neural network comprising at least one initial series of layers, the at least one initial series of layers comprising an initial depthwise convolutional layer, then a first batch of normalized layers, then an initial saturated rectified linear unit layer, and then an initial two-dimensional convolutional layer; and The first batch of normalization layers of the initially trained artificial neural network are fused with the initial depth convolutional layer to obtain a first trained artificial neural network.

12. The non-transitory computer readable medium of claim 11, wherein the at least one initial series of layers further comprises a second batch of normalization layers subsequent to the initial two-dimensional convolutional layers, and the non-transitory computer readable medium comprises further computer instructions that, when executed by the processor, cause the processor to perform the following steps: The second batch of normalization layers are fused with the initial two-dimensional convolutional layer to obtain a first trained artificial neural network.

13. A computer system for compiling a first trained artificial neural network, the first trained artificial neural network comprising at least one series of layers, the at least one series of layers comprising a depthwise convolutional layer, then a saturated rectified linear unit layer, and then a two-dimensional convolutional layer, the computer system comprising: a non-transitory memory storage device comprising computer instructions; as well as a processor in communication with the non-transitory memory storage device, wherein the processor executes the computer instructions to: Performing balancing between the depth convolution layer and the two-dimensional convolution layer; replacing the saturated rectified linear unit layer with an adaptive channel pruning layer to obtain an artificial neural network with a modified topology; performing tensor quantization on the layer of the artificial neural network having the modified topology to obtain a quantized artificial neural network having the modified topology; and The quantized artificial neural network with the modified topology is compiled to generate a computer program, wherein the computer program includes compilation instructions, and when the computer program is executed by a computer, the compilation instructions enable the computer to implement the quantized artificial neural network with the modified topology.

14. The computer system of claim 13, wherein the adaptive channel pruning layer is adapted according to a saturation value defined for the saturated rectified linear unit layer.

15. The computer system of claim 14, wherein the adaptive channel pruning layer is configured to prune each channel of an activation tensor generated by the deep convolutional layer between a minimum value and a maximum value, the minimum value being equal to 0, the maximum value being calculated by applying the following formula: where max_val is the maximum value, X is the saturation value defined by the Saturated Rectified Linear Unit layer, and where r i (1) corresponds to a first range of values ​​for channel i of the first weight tensor of the depthwise convolutional layer, and r i (2) A second range of values ​​corresponding to channel i of the second weight tensor of the two-dimensional convolutional layer.

16. The computer system of claim 13, wherein the processor executes additional computer instructions to perform bias absorption between the depthwise convolutional layer and the two-dimensional convolutional layer.

17. The computer system of claim 13, wherein the processor executes the computer instructions to: obtaining an initially trained artificial neural network, the initially trained artificial neural network comprising at least one initial series of layers, the at least one initial series of layers comprising an initial depthwise convolutional layer, then a first batch of normalized layers, then an initial saturated rectified linear unit layer, and then an initial two-dimensional convolutional layer; and The first batch of normalization layers of the initially trained artificial neural network are fused with the initial depth convolutional layer to obtain a first trained artificial neural network.

18. The computer system of claim 17, wherein the at least one initial series of layers further comprises a second batch of normalization layers, the second batch of normalization layers following the initial two-dimensional convolutional layers, and the processor executes further computer instructions to: The second batch of normalization layers are fused with the initial two-dimensional convolutional layer to obtain a first trained artificial neural network.

19. A microcontroller comprising: a non-transitory memory storage device, the non-transitory memory storage device including compiled instructions; as well as a processor in communication with the non-transitory memory storage device, wherein the compiled instructions include a quantized artificial neural network, the quantized artificial neural network having a modified topology, the quantized artificial neural network being generated from a first trained artificial neural network, the first trained artificial neural network comprising at least one series of layers, the at least one series of layers comprising a depthwise convolutional layer, then a saturated rectified linear unit layer, and then a two-dimensional convolutional layer, the quantized artificial neural network having the modified topology being generated by: Performing balancing between the depth convolution layer and the two-dimensional convolution layer; Replacing the saturated rectified linear unit layer with an adaptive channel pruning layer to obtain an artificial neural network with the modified topology; performing tensor quantization on the layer of the artificial neural network having the modified topology to obtain a quantized artificial neural network having the modified topology; as well as The quantized artificial neural network with the modified topology is compiled to generate the compiled instructions, which, when executed by the microcontroller, enable the microcontroller to implement the quantized artificial neural network with the modified topology.

20. The microcontroller of claim 19, further comprising a neural network processing circuit configured to run the quantized artificial neural network having the modified topology.

Images