Method of training convolutional neural network comprising nodes arranged in layers and pruned masks

Abstract

The invention relates to a computer-implemented method for training a convolutional neural network, the convolutional neural network comprising nodes arranged in layers and a prune mask, the method comprising the steps of:-providing at least one marked training data set (S10); -initializing the convolutional neural network (S12); -passing the training data through the convolutional neural network (S14); -calculating a loss function (S16) and comparing the loss function with the labels of the training data; and-minimizing the loss function by back propagation (S18), including determining a gradient of the loss function; wherein passing the training data through the delete mask comprises multiplying the structure si of the input of the delete mask by a delete parameter Mi, the delete parameter Mi for the structure si being 0 or 1; wherein the mask is approximately pruned by an approximation function during back propagation.

Description

Technical Field

[0001] The present invention relates to the field of training convolutional neural networks using structured pruning. This means that complete channels, i.e. input channels, output channels, filters or other smaller structures, such as half a filter, are removed from the convolutional layers of the convolutional neural network. Background Art

[0002] A convolutional neural network (CNN) is a special type of neural network. CNN is a popular architecture for processing data with spatial structure, such as images or audio signals.

[0003] The fundamental operation in CNN is convolution. Convolution is a mathematical operator that superimposes one function on another to create a new function. In the context of CNN, convolution refers to applying a filter to the input data to extract features.

[0004] CNNs consist of several layers, the most important of which are convolutional layers. In these convolutional layers, filters or so-called "kernels" are used to extract local features from the input data. Each filter consists of a small weight matrix that is pushed onto the input data to perform a convolution. This convolution produces an activation map that represents the filter's response to various features in the input signal.

[0005] Neural network compression is a key tool for enabling deployment in embedded devices. One of the most commonly used compression methods is pruning, which consists of removing redundant or unnecessary elements from the neural network architecture. Pruning can be performed in both unstructured and structured ways.

[0006] Unstructured pruning removes individual weights, which results in a high pruning ratio (also called compression ratio). However, such an approach causes irregular memory access and computation, which ultimately does not bring any practical resource improvement in embedded hardware platforms.

[0007] On the other hand, structured pruning can effectively reduce computational resources by removing complete neural network structures such as channels. Traditional structured pruning methods eliminate structures based on importance metrics (such as L1 norm). This pruning process is performed for each layer as follows: the importance of structural aspects is iteratively calculated while monitoring the accuracy of the neural network until the accuracy tolerance or the maximum pruning ratio is reached.

[0008] Another structural pruning selection method is presented in Channel pruning for accelerating very deep neural networks (CoRR, abs / 1707.06168, 2017) by He et al. In this method, at each layer, less important structures are eliminated by LASSO regression. Then, the pruned layers are fine-tuned. These methods are quite time-consuming because fine-tuning needs to be performed every time a layer is pruned. Additionally, this is not optimal when some layers are highly sensitive to pruning and should not even be compressed at all, because it considers the same pruning ratio for all layers.

[0009] More modern approaches consider training a so-called pruning mask. The mask has elements that are either 0 or 1, and each element is multiplied by the corresponding structure of the layer to be pruned. In Discrete model compression with resource constraint for deep neural networks (IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1896–1905, 2020) by Gao et al., a method is proposed in which the pruning mask is represented as a random vector with values ​​(0, 1) that is trained using stochastic gradient descent (SGD) when the neural network weights are frozen. Training is constrained by the target number of floating-point operations (FLOPs), and the regularization loss is the logarithmic loss R(x, y) = log(|xy|+1), where x is the current number of FLOPs and y is the target number of FLOPs. During mask training, this loss is added to the original cost function C of the target neural network model. After the pruning mask has been trained, the neural network model is pruned with the resulting mask and fine-tuned for some rounds to improve accuracy.

[0010] In Disentangled Differentiable Network Pruning by Gao et al. (European Conference on Computer Vision, 2022), the authors determine the pruning mask of the target neural network model by using a smaller neural network model composed of gated recurrent units (GRU) or fully connected layers (FC) to learn the importance of structure and the width (or number of structures) of the layers. The pruning mask is thus represented by a Smoothstep function to allow soft relaxation. The weights of the target neural network are frozen and the smaller neural network is trained. After the smaller neural network model has been trained and the pruning mask has been obtained, the target neural network is then fine-tuned to improve accuracy.

[0011] The first problem presented by current structure pruning methods is that they require a first optimization phase to obtain the pruning mask or determine the structure to be pruned, and then a second optimization phase is required to fine-tune the pruned model. Although some methods use SGD to estimate the pruning mask, the cost functions they propose do not allow for simultaneous training of the pruning mask and the target neural network model. The second problem is that due to the sequential nature of the pruning methods, it is impossible to perform joint optimization of pruning and other compression methods that require fine-tuning (e.g., quantization).

[0012] The problem to be solved by the present invention is to propose a method for structured pruning that does not require any modification of the cost function and allows to train both the neural network weights and the masks simultaneously to significantly reduce the optimization time.

[0013] This problem is solved by the subject matter of the independent claims. Summary of the invention

[0014] According to a first aspect, the present invention relates to a computer-implemented method for training a convolutional neural network (CNN) comprising nodes arranged in layers and a pruning mask. The method comprises the following steps:

[0015] - providing at least one labeled training data set;

[0016] - initializing a convolutional neural network comprising at least one convolutional layer and a pruning mask following said convolutional layer;

[0017] - Pass the training data through the convolutional neural network;

[0018] - Compute a loss function and compare the loss function to the labels of the training data to quantify the convolutional neural network process; and

[0019] -Minimizing a loss function via back-propagation, including determining the gradient of the loss function to be used to determine the direction in which to optimize the nodes of the convolutional neural network.

[0020] Passing the training data through the pruning mask includes pruning the masked input structure s i Multiply by the pruning parameter M i , the structure of i The pruning parameter M i is either 0 or 1. During back-propagation, the pruning mask is approximated by an approximation function.

[0021] The training data is provided in sets. For each set, the CNN generates an output. The output may include statements about the input data, its representation, etc. In addition, the provision of training data may include any kind of pre-processing for processing the data by the CNN. This may include, but is not limited to, scaling input images, transforming wave-based data such as audio data, etc.

[0022] Labeling the training data is necessary as the labels will indicate whether the CNN has processed the data correctly. If the data has not been processed such that the CNN has found the correct labels as given by the provided training dataset, the parameters of the CNN are adjusted.

[0023] During initialization, the architecture of the CNN is defined. This includes the number of convolutional layers, the number of pooling layers, the number of fully connected layers, and the number of activation functions. Then, the weights and biases of each neuron or node are initialized, which means that they get a starting value that may or may not be adjusted during training.

[0024] When training data is passed through a CNN, the data passes through each layer one by one. The first layer to be passed through is a convolutional layer, which is used to identify points of interest (POI) or regions of interest (ROI) in the data. If the input data is image data, the convolutional layer can identify a group of pixels to be analyzed by the CNN in high detail.

[0025] As the data is passed through each layer, each operation, such as convolution, pooling, or activation function, is applied to the data. Then, an activation map is generated. For example, each filter (convolution layer) detects a specific feature or pattern in the input image. The convolution operation involves sliding the filter spatially across the input, computing element-wise multiplications and sums. This process produces a feature map that highlights the areas in the input image where the filter's features were detected.

[0026] Activation maps capture learned representations at different levels of abstraction within a CNN. As data passes through successive convolutional layers, activation maps become increasingly complex, representing higher-level features. Early layers capture low-level features such as edges and textures, while deeper layers learn more complex features such as shapes, objects, or semantic concepts.

[0027] Activation maps provide valuable insights into how CNNs process and understand input data by visualizing and explaining the network’s feature extraction capabilities.

[0028] The step of calculating the loss function and comparing it to the labels of the training data to quantify the convolutional neural network process is part of the backward pass or backpropagation of the training process. Therefore, in CNN, the loss function measures the difference between the predicted output of the network and the true labels of the training data. It quantifies how good the network is in its ability to correctly classify or predict the target output.

[0029] The choice of an appropriate loss function depends on the specific task and the nature of the problem being solved. Commonly used loss functions in CNNs can include mean squared error (MSE) or cross entropy loss.

[0030] The MSE loss function calculates the mean squared difference between the predicted label and the true label. It is often used for regression problems. The cross entropy loss measures the difference between the predicted class probability and the true label. It is often used for classification tasks.

[0031] Choosing an appropriate loss function is crucial because it affects the learning behavior of the network and the optimization process. Different loss functions have different properties and are more suitable for specific tasks or data distributions. When choosing an appropriate loss function to train a CNN, it is important to consider the characteristics of the problem at hand.

[0032] During backpropagation, the loss function is used to calculate the gradient of the network parameters with respect to the loss. This gradient provides information about the direction and magnitude of the adjustments needed to minimize the loss. The gradient is then backpropagated through the network, allowing the weights and biases to be iteratively updated using an optimization algorithm such as, but not limited to, stochastic gradient descent (SGD).

[0033] The goal of training a CNN is to minimize a loss function by iteratively adjusting the network parameters. As the network is trained on more data and the optimization progresses, the loss function should decrease, indicating an improvement in the network's predictive performance.

[0034] Only consider the filter dimension s out , the proposed invention requires a truncation mask M, which is of length s out The mask value can be either 0 or 1. Such structured pruning can then be formulated as:

[0035] Y q,pruned =Y q **M q

[0036] Where q denotes the qth structure. The cost function used to train the target CNN model is also used to optimize the pruning mask.

[0037] For back-propagation of training, the gradient of this discrete function does not exist, and so an approximation has to be applied instead.

[0038] A possible approximation of the rounding function could be:

[0039]

[0040] Another possibility is to compute a steeper sigmoid, and remove the rounding function during backpropagation, which gives:

[0041]

[0042] where γ regulates the steepness (larger γ results in a steeper function).

[0043] The closer the approximation is to the actual rounding function, the better the parameters will be updated after backpropagation.

[0044] By providing a pruning mask that can be trained by the same cost function as the convolutional layer and an approximation of the discrete function of the pruning mask, the CNN can be trained without modifying the cost function, and therefore, the pruning mask can be trained simultaneously with the rest of the CNN to significantly reduce the optimization time. Therefore, the present invention solves this problem.

[0045] In one embodiment, the deletion parameter M i According to the auxiliary vector Z i Determine, where the auxiliary vector Z i are the trainable parameters of the convolutional neural network, and the approximation function is Z i function.

[0046] As mentioned before, the mask can only have the value 0 or 1, which are discrete values ​​and thus may not be directly trainable. To achieve mask optimization, an auxiliary vector Z can be used. The vector Z can then be viewed as an importance ranking of each filter q. The function max(Z) can be defined as the most important filter and min(Z) as the least important filter, respectively.

[0047] In another embodiment, the pruning parameter is derived from rounding the sigmoid function, where the auxiliary vector Z i is the input of the sigmoid function.

[0048] The correlation between M and Z can be expressed, for example but not limited to:

[0049] M = round(σ(Z))

[0050] Where 0<∑ i M i ≤β*ch out

[0051] where Z is a trainable parameter, σ(·) represents the sigmoid function, and β is The compression ratio is defined as the number of weights before pruning divided by the number of weights after pruning. The sigmoid function is the probability of interpolation between [0, 1] of the trainable vector Z.

[0052] In one embodiment, the auxiliary vector Z is initialized with a value of 0 or close to 0. i Values ​​close to 0 may include, for example, values ​​in the range of 1e-4 or 1e-5.

[0053] It is useful to initialize Z with a value at or close to 0 so that the mask M i Converge faster to M = 0 or M = 1. This automatically distinguishes important filters Z>0 from unimportant filters Z<0 by manipulating Z according to its direct impact on the main CNN cost function.

[0054] In one embodiment, all pruning parameters M i The sum of is greater than 0. In other words, the following conditions are met:

[0055]

[0056] This constraint is satisfied by choosing the sum of all pruning parameters to be greater than 0, ensuring that the most important filter will never be pruned. For example, the constraint can be implemented by searching and replacing the largest element in Z, and if that element is less than 0, replacing it with a very small value such as 1e-4.

[0057] In one embodiment, all pruning parameters M i The sum of is less than or equal to the number of output structures s of the convolutional layer out Divide by the compression ratio of the pruning mask.

[0058] This constraint ensures that at least k filters are pruned (k>0; to ensure minimum compression). If any of the min-k elements of Z are non-negative, a negative offset is applied so that the resulting mask is equal to 0. By selecting the min-k elements, it is ensured that the least important filter is the one that is pruned.

[0059] In an alternative embodiment, all pruning parameters M i The sum of is less than or equal to the factor that corresponds to the hardware limit of the specified operating system.

[0060] By including the hardware limitations of a specific operating system, CNNs can be trained more efficiently for or even on a specific system.

[0061] In one embodiment, a convolutional neural network is designed for processing image or audio data, and is particularly implemented as an eye tracking application.

[0062] Training for eye tracking applications may be structured, for example but not limited to, as follows.

[0063] In the first step, eye tracking data is collected using a specialized eye tracking device or sensor. This data typically includes information about the user's gaze position or eye movements.

[0064] In the second step, the collected eye tracking data is pre-processed to remove noise and artifacts. This may involve filtering, calibration, and normalization techniques to ensure data quality.

[0065] Then, in the third step, the pre-processed eye tracking data is combined with the corresponding input images or visual stimuli. These input images are used as input to the CNN, while the eye tracking data serves as the ground truth or label.

[0066] Next, in the fourth step, the CNN architecture is designed, which involves choosing appropriate layers, filter sizes, pooling operations, and activation functions. The architecture should be tailored to the eye tracking task at hand, taking into account factors such as input image resolution, eye movement patterns, and desired accuracy.

[0067] A CNN is trained using the created dataset. The input image is fed into the network and the predicted gaze position or eye movement is compared with the ground truth labels. This is the layer in the CNN where pruning can be applied. Regions of Interest (ROI) or Points of Interest (POI) can be selected where other areas of the input data are pruned. The difference between the predicted and actual eye movement is used to calculate the loss function.

[0068] In the next sixth step, back propagation is performed to update the weights and biases of the CNN based on the calculated loss. The gradient of the loss function with respect to the network parameters is calculated and the weights are adjusted using an optimization algorithm. The pruning mask can be approximated by a suitable function such as the one mentioned above.

[0069] The trained CNN is then evaluated on a separate validation dataset to measure its performance. Hyperparameters such as learning rate, batch size, and network architecture can be tuned to improve the accuracy and generalization ability of the model.

[0070] Once a CNN is trained and validated, it can be used to make eye tracking predictions on new, unseen data. Input images are fed into the trained network, and the CNN generates predictions for gaze locations or eye movements. The output of the CNN can be post-processed and visualized to provide meaningful insights. This may involve mapping the predicted gaze locations onto the input image, or generating heat maps to highlight areas of visual attention.

[0071] Finally, the performance of the eye-tracking CNN can be further improved through an iterative cycle of training, validation, and fine-tuning. This may involve collecting more data, refining the network architecture, or incorporating additional techniques (such as data augmentation or regularization). However, this step can be shortened thanks to the trainable pruning mask layer.

[0072] In another aspect, the invention relates to a computer program comprising a program code for executing a computer-implemented method as described above, when the computer program is run on a computer.

[0073] In a further aspect, the invention relates to a computer-readable data carrier comprising program code of a computer program which, if run on a computer, executes the computer-implemented method as described above.

[0074] In yet another aspect, the present invention relates to a system for training a convolutional neural network using pruning masks, wherein the system comprises components for running the computer-implemented method as described above.

[0075] The described embodiments and further developments can be combined with one another as desired.

[0076] Further possible embodiments, further developments and implementations of the invention also include combinations of features of the invention which have not been explicitly mentioned and which have been described above or below with respect to the exemplary embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings are intended to provide a further understanding of the embodiments of the present invention. They illustrate the embodiments and, together with the description, are used to explain the principles and concepts of the present invention.

[0078] Other embodiments and the many advantages mentioned will be apparent with reference to the accompanying drawings.The elements shown in the drawings are not necessarily shown to scale relative to each other.

[0079] The figures show:

[0080] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0081] Figure 2 A schematic diagram illustrating the principles of the present invention;

[0082] Figure 3 A sigmoid function, such as may be used in the present invention; and

[0083] Figure 4 Different functions as they may be used according to the invention.

[0084] In the various figures of the drawings, unless otherwise indicated, like reference numerals indicate identical or functionally identical elements, components or assemblies. DETAILED DESCRIPTION

[0085] Figure 1 A schematic diagram of a method according to an embodiment of the present invention is shown.

[0086] In a first step S10, training data is collected and preprocessed to remove, for example, noise and artifacts. This may involve filtering, calibration and normalization techniques to ensure data quality. The preprocessed training data is labeled, which is required to find the ground truth or identify the correct processing of the CNN on the data.

[0087] In step S12, the CNN architecture is designed, which involves selecting appropriate layers, filter sizes, pooling operations, and activation functions. The design of the CNN also includes initializing the values, weights, and biases of each node within the CNN.

[0088] In step S14, the training data is passed through the CNN. The input images are fed into the network and the network finds a prediction for each training set. When compared with the labels, the prediction will be more or less correct.

[0089] In step S16, the difference between the predicted label and the actual label is used to calculate the loss function. The loss function is a core element of the training process because it is used to refine the weights, biases, and values ​​of each node in the network.

[0090] In step S18, based on the calculated loss, back propagation is performed to update the weights and biases of the CNN. The gradient of the loss function with respect to the network parameters is calculated, and the weights are adjusted using an optimization algorithm. The pruning mask can be approximated by a suitable function (such as a sigmoid function).

[0091] Steps S14 to S18 may be repeated until the loss function has a sufficiently low value.

[0092] The trained CNN is then evaluated on a separate validation dataset to measure its performance. Hyperparameters such as learning rate, batch size, and network architecture can be tuned to improve the accuracy and generalization ability of the model.

[0093] Once a CNN is trained and validated, it can be used for its original purpose on new, unseen data. Input data is fed into the trained network, and the CNN generates predictions based on its policy. The output of the CNN can be post-processed and visualized to provide meaningful insights. This may involve mapping the output to relevant parameters, or generating heat maps to highlight areas of visual interest.

[0094] Finally, the performance of the CNN can be further improved through an iterative cycle of training, validation, and fine-tuning. This may involve collecting more data, refining the network architecture, or incorporating additional techniques (such as data augmentation or regularization). However, this step can be shortened by applying a pruning mask that can be trained with the remaining layers of the CNN.

[0095] Figure 2 A schematic diagram of a pruned CNN 10 is shown. The CNN may include multiple layers 12, each layer having multiple structures 14. In each layer 12, each structure 14 may be processed.

[0096] However, one of the layers 12 shown is a pruning mask 16. The structure 14 of the pruning mask 16 is arranged to multiply the output from the layer 18 preceding the pruning mask 16 by the pruning parameter M. Thus, the structure s i The output of is multiplied by the pruning parameter M corresponding to the structure 14 i .

[0097] Deletion parameter M i Can be 1 if layer 18 is used as input to layer 20 beyond the pruning mask, to pass the output of the structure. Otherwise, the pruning parameter can be 0 to prune structure 14. Figure 2 In FIG. 1 , four structures 14 are shown as being pruned, while another four structures 14 are shown passing the structure output of layer 18 to layer 20 as structure input.

[0098] Figure 3 A graph of the sigmoid function is shown. The function shown is:

[0099]

[0100] When rounding, the function round(sig(t)) will give the value 1 for each t greater than or equal to 0, and the value 0 for each t less than 0. When t is replaced by an auxiliary vector Z, it may give the value 1 or 0 depending on the auxiliary vector Z.

[0101] Figure 4 Two functions of f(Z) are shown, where Z is an auxiliary vector. One of the functions f(Z) is a rounding function round(Z) that gives a value of 1 for positive Z and a value of 0 for negative Z. This function can be used to determine the pruning parameters for which Z is the corresponding parameter to be trained in the CNN.

[0102] The rounding function has no gradient because it is not a continuous function. At Z=0, it switches from 0 to 1, which results in a difference of infinity. But for training Z, we have to derive the gradient to check whether Z has to go up or down.

[0103] The gradient function for the rounding function is approximated by an approximation function, which can be like Figure 4 As shown in approx (Z′) is a continuous function from which the gradient can be derived.

[0104] The combination of two functions implements a trainable pruning mask for CNN. One function is used for forward propagation, and the other function can be applied during backpropagation.

Claims

1. A computer-implemented method for training a convolutional neural network comprising nodes arranged in layers and a pruning mask, The method comprises the following steps: - providing at least one labeled training data set (S10); - Initializing a convolutional neural network (S12), which includes at least one convolutional layer and a pruning mask after the convolutional layer; - passing the training data through a convolutional neural network (S14); - calculating a loss function (S16) and comparing the loss function with the labels of the training data to quantify the convolutional neural network process; and - minimizing the loss function by back propagation (S18), including determining the gradient of the loss function for determining the direction of optimizing the convolutional neural network nodes; where passing the training data through the pruning mask includes pruning the input structure s of the mask i Multiply by the pruning parameter M i , used for the structure of i The pruning parameter M i is 0 or 1; where the pruning mask is approximated by an approximation function during back-propagation.

2. The computer-implemented method of claim 1, wherein the pruning parameter M i is the auxiliary vector Z i Determined, where the auxiliary vector Z i are trainable parameters of the convolutional neural network, and where the approximation function is Z i function.

3. The computer-implemented method of claim 2 , wherein the pruning parameter is derived from rounding a sigmoid function, wherein the auxiliary vector Z i is the input of the sigmoid function.

4. The computer-implemented method of claim 3, wherein the auxiliary vector Z i Initialize with a value of 0 or close to 0.

5. The computer-implemented method according to claim 1 , wherein all pruning parameters M i The sum of is greater than 0.

6. The computer-implemented method according to claim 1, wherein all pruning parameters M i The sum of is less than or equal to the number of output structures s of the convolutional layer out divided by the compression ratio of the pruning mask, or All the deleted parameters M i The sum of is less than or equal to the factor that corresponds to the hardware limit of the specified operating system.

7. The computer-implemented method according to one of the preceding claims, wherein the convolutional neural network is designed for processing image or audio data, in particular implemented as an eye tracking application.

8. Computer program comprising a program code for executing the computer-implemented method according to one of the preceding claims, when said computer program is run on a computer.

9. Computer-readable data carrier comprising program code of a computer program for executing the computer-implemented method according to one of claims 1 to 7 if the computer program is executed on a computer.

10. A system for training a convolutional neural network using pruning masks, wherein the system comprises means for running a computer-implemented method according to one of claims 1 to 7.

Images