Method for training a convolutional neural network comprising nodes arranged in layers and a clipping mask

Abstract

The invention relates to a computer-implemented method for training a convolutional neural network comprising nodes arranged in layers and a clipping mask, wherein the method comprises the following steps: - Provide at least a set of labeled training data (S10); - Initializing the convolutional neural network (S12); - Routing the training data through the convolutional neural network (S14); - Calculating a loss function (S16) and comparing the loss function with labels of the training data; and - Minimizing the loss function by backward propagation (S18), including determining a gradient of the loss function; where guiding the training data through the clipping mask multiplies the structures s i the input of the clipping mask with a clipping parameter M i includes, where the clipping parameter Mi for the s i the structure is 0 or 1; where the clipping mask is approximated by an approximation function during backward propagation.

Description

[0001] The invention relates to the field of training a convolutional neural network using structured pruning. This means that entire channels, i.e., input channels, output channels, filters, or other smaller structures, e.g., half of a filter, are removed from a convolutional layer of a convolutional neural network.

[0002] A CNN (convolutional neural network) is a special type of neural network. CNNs are a popular architecture for processing data with a spatial structure, such as images or audio signals.

[0003] The basic operation in a CNN is convolution. A convolution is a mathematical operator that superimposes one function on another to generate a new function. In the context of CNNs, convolution refers to the application of filters to input data to extract features.

[0004] A CNN consists of several layers, the most important of which are the convolutional layers. These convolutional layers use filters or kernels to extract local features from the input data. Each filter consists of a small matrix of weights that is superimposed over the input data to perform a convolution. This convolution produces an activation map that represents the filter's response to various features in the input signal.

[0005] Neural network compression is a key tool for enabling deployment in embedded devices. One of the most widely used compression techniques is pruning, which involves removing redundant or unnecessary elements from the neural network architecture. Pruning can be performed in both unstructured and structured ways.

[0006] Unstructured pruning removes individual weights, resulting in a high pruning ratio, also known as compression ratio. However, such an approach leads to irregular memory accesses and computations, ultimately providing no practical resource improvements in embedded hardware platforms.

[0007] Structured pruning, on the other hand, can effectively reduce computational resources by removing entire neural network structures, such as channels. Traditional structured pruning methods remove structures according to a significance metric, such as the L1 norm. This pruning process is performed for each layer as follows: the structure-wise significance is iteratively calculated while monitoring the neural network's accuracy until the accuracy tolerance or maximum pruning ratio is reached.

[0008] Another structure pruning selection approach is presented in He et al., "Channel pruning for accelerating very deep neural networks," CoRR, abs / 1707.06168, 2017. In this method, less significant structures are removed in each layer using LASSO regression. Then, the pruned layer is fine-tuned. These approaches are relatively time-consuming because fine-tuning must be performed each time a layer is pruned. Furthermore, this is suboptimal because it considers the same pruning ratio for all layers, while there are some layers that are highly sensitive to pruning and should not be compressed at all.

[0009] More modern methods consider training a so-called clipping mask. This mask has elements that are either 0 or 1, and each element multiplies the corresponding structure of the layer to be clipped. In Gao et al., Discrete model compression with resource constraint for deep neural networks, IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1896–1905, 2020, a method is proposed in which the clipping mask is represented as a stochastic vector with values ​​(0, 1), trained using SGD (stochastic gradient descent) while the neural network weights are frozen. Training is constrained by the target number of FLOPs (floating-point operations), and the regularization loss is a logarithmic loss R(x, y) = log(|x - y| + 1), where x is the current number of FLOPs and y is the target number of FLOPs.This loss is added to the original cost function C of the target neural network model during mask training. After the clipping mask is trained, the neural network model is clipped with the resulting mask and fine-tuned for several epochs to improve accuracy.

[0010] In Gao et al., "Disentangled differentiable network pruning," In European Conference on Computer Vision, 2022, the authors determine the pruning mask of a target neural network model by using a smaller neural network model consisting of GRUs (gated recurrent units) or fully connected layers (FC) to obtain structure-wise significance and layer-wise width (or number of structures). The pruning mask is represented with a smoothstep function to allow for soft relaxation. The weights of the target neural network are frozen, and a smaller neural network is trained. After training the smaller neural network model and obtaining the pruning mask, the target neural network is then fine-tuned to improve accuracy.

[0011] The first problem posed by current structure pruning methods is that they require an initial optimization phase to obtain the clipping mask or to determine the structures to be clipped, and then a second optimization phase to fine-tune the clipped model. While some methods use SGD to estimate the clipping mask, their proposed cost functions do not allow for concurrent training of the clipping mask and the target neural network model. The second problem is that, due to the sequential nature of clipping methods, it is not possible to perform joint optimization of clipping and other compression methods that require fine-tuning, such as quantization.

[0012] The problem to be solved by the invention is to propose a method for structured pruning that does not require any modification of the cost function and allows the training of neural network weights and masks simultaneously in order to significantly reduce the optimization time.

[0013] The problem is solved by the subject matter of the independent claims. Disclosure of the invention

[0014] According to a first aspect, the invention relates to a computer-implemented method for training a CNN (convolutional neural network) comprising layered nodes and a clipping mask. The method comprises the following steps: - Providing at least one set of labelled training data; - Initializing the convolutional neural network comprising at least one convolutional layer and a clipping mask behind the convolutional layer; - Passing the training data through the convolutional neural network; - Calculating a loss function and comparing the loss function with labels of the training data to quantify the convolutional neural network process; and - Minimizing the loss function by backpropagation, including determining a gradient of the loss function to determine a direction for optimizing the nodes of the convolutional neural network.

[0015] Passing the training data through the clipping mask involves multiplying the structures s i entering the clipping mask with a clipping parameter M i , where the clipping parameter M i for the s i of the structure is 0 or 1. The clipping mask is approximated by an approximation function during backpropagation.

[0016] The training data is provided in sets. For each set, the CNN generates an output. The output may include a statement about the input data, a characterization of the input data, or the like. Furthermore, providing the training data may include any type of preprocessing for processing the data by the CNN. This may include, but is not limited to, scaling an input image, transforming wave-based data such as audio data, etc.

[0017] Labeling the training data is crucial because the labels indicate whether the CNN processed the data correctly or not. If the data was not processed in such a way that the CNN found the correct label, as determined by the provided training data sets, the CNN's parameters are adjusted.

[0018] During initialization, the architecture of the CNN is defined. This includes the number of convolutional layers, pooling layers, fully connected layers, and activation functions. Then, the weights and biases of each neuron or node are initialized, meaning they receive a starting value that may or may not be adjusted during training.

[0019] When the training data is passed through the CNN, the data passes through each layer individually. The first layers to be passed through are convolutional layers for identifying points of interest (POIs) or regions of interest (ROIs) in the data. If the input data is image data, the convolutional layers can identify a group of pixels to be analyzed in high detail by the CNN.

[0020] As the data passes through each layer, each operation is applied to the data, such as convolutions, pooling, or activation functions. Then, an activation map is generated. For example, each filter (convolutional layer) detects specific features or patterns in the input image. The convolution operation involves spatially shifting the filter over the input, calculating element-wise multiplications and summations. This process produces a feature map that highlights the regions of the input image where the filter's features are detected.

[0021] The activation maps capture the learned representations at different levels of abstraction in the CNN. As the data passes through successive convolutional layers, the activation maps can become increasingly complex, representing higher-level features. The earlier layers capture low-level features, such as edges and textures, while deeper layers learn more complex features, such as shapes, objects, or semantic concepts.

[0022] Activation maps provide valuable insights into how a CNN processes and understands the input data by visualizing and interpreting the network's feature extraction capabilities.

[0023] The step of calculating a loss function and comparing it to the labels of the training data to quantify the convolutional neural network process is the backpass, or backpropagation, part of the training process. In CNNs, the loss function thus measures the discrepancy between the network's predicted output and the true labels of the training data. It quantifies how well the network performs in terms of its ability to correctly classify or predict the target outputs.

[0024] The choice of a suitable loss function depends on the specific task and the nature of the problem being addressed.

[0025] Commonly used loss functions in CNNs can include an MSE (mean square error) or a cross-entropy loss.

[0026] The MSE loss function calculates the mean squared difference between the predicted and true labels. It is often used for regression problems. Cross-entropy loss measures the dissimilarity between the predicted class probabilities and the true labels. It is commonly used for classification tasks.

[0027] Choosing an appropriate loss function is crucial because it influences the learning behavior and optimization process of the network. Different loss functions have different properties and are better suited for specific tasks or data distributions. It is important to consider the properties of the given problem when selecting the appropriate loss function for training a CNN.

[0028] During backpropagation, the loss function is used to calculate the gradient of the network's parameters with respect to the loss. This gradient provides information about the direction and amount of adjustments necessary to minimize the loss. The gradient is then backpropagated through the network, allowing iterative updating of weights and biases using optimization algorithms such as, but not limited to, SGD (stochastic gradient descent).

[0029] The goal of training a CNN is to minimize the loss function by iteratively tuning the network's parameters. As the network is trained on more data and optimization progresses, the loss function should decrease, indicating an improvement in the network's predictive performance.

[0030] If you only use the filter dimensions s outconsidered, the proposed invention requires a clipping mask M, which is a vector of length s out Mask values ​​can be either 0 or 1. Such a structured clipping can then be formulated as Yq,pruned=Yq*Mq where q represents the qth structure. The cost function used to train the target CNN model is also used to optimize the clipping mask.

[0031] For the backpropagation of the training, no gradient of this discrete function exists, and therefore an approximation must be used instead.

[0032] A possible approximation of the rounding function would be roundapprox(Z′)=Z′−sin(2πZ′)2π

[0033] Another possibility is to calculate a steeper sigmoid and remove the rounding function during backpropagation, which gives: σsteep=11+e(−γz) where γ controls the steepness (larger γ leads to steeper function).

[0034] The closer the approximation comes to the actual rounding function, the better the parameters are updated after backpropagation.

[0035] By providing a clipping mask that can be trained using the same cost function as the convolutional layers and an approximation function for the discrete function of the clipping mask, the CNN can be trained without modifying the cost function, thus providing a clipping mask that can be trained concurrently with the rest of the CNN to significantly reduce optimization time. Therefore, the invention solves this problem.

[0036] In one embodiment, the clipping parameter M i from a helper vector Z i where the helper vector Z iis a trainable parameter of the convolutional neural network and where the approximation function is a function of Z i is.

[0037] As mentioned above, the mask can only have values ​​of 0 or 1, which are discrete values ​​and therefore cannot be trained directly. To enable mask optimization, a helper vector Z can be used. The vector Z can be considered a ranking of each filter q. The function max(Z) can be defined as the most important filter, and min(Z) as the least important filter.

[0038] In another embodiment, the clipping parameter is derived from rounding a sigmoid function, where the helper vector Z i is the input of the sigmoid function.

[0039] The dependence between M and Z can be expressed, for example but not limited to, as follows: M=round(σ(Z)) with:0<∑iMi≤β∗chout where Z is a trainable parameter and σ(·) represents the sigmoid function and β1 compression rate The compression ratio is defined as the number of weights before pruning divided by the number of weights after pruning. The sigmoid function is a way to interpolate between [0,1] of the trainable vector Z.

[0040] In one embodiment, the helper vector Z i initialized with a value of 0 or close to 0. Values ​​close to 0 can include values ​​in the range of, for example, 1e-4 or 1e-5.

[0041] Usefully, by initializing Z with a value of 0 or close to 0, the mask M iconverges faster to either M = 0 or M = 1. This automatically distinguishes important filters Z > 0 from unimportant filters Z < 0 by steering Z according to its direct impact on the main CNN cost function.

[0042] In one embodiment, the sum of all clipping parameters M i greater than 0. In other words, the condition 0<∑iMi fulfilled.

[0043] Choosing the sum of all clipping parameters greater than 0 to satisfy this constraint ensures that the most important filters are never clipped. For example, the constraint can be implemented by finding and replacing the largest element in Z and replacing it with a very small value such as 1e-4 if that element is less than 0.

[0044] In one embodiment, the sum of all clipping parameters M iless than or equal to the number of output structures of the convolutional layer s out , divided by the compression ratio of the clipping mask.

[0045] This constraint guarantees that at least k filters are clipped (k>0; guaranteed minimum compression ratio). If any min-k element of Z is non-negative, a negative offset is applied so that the resulting mask is 0. Selecting the min-k elements ensures that the least important filters are the ones clipped.

[0046] In an alternative embodiment, the sum of all clipping parameters M i less than or equal to a factor corresponding to the hardware limitations of the target operating system.

[0047] By incorporating hardware limitations of the target operating system, the CNN can be trained more efficiently for the target system, or even on the target system.

[0048] In one embodiment, the convolutional neural network is designed for processing image or audio data and is particularly implemented as an eye tracking application.

[0049] For example, but not limited to, training for an eye tracking application could be structured as follows.

[0050] In a first step, eye tracking data is collected using specialized eye tracking devices or sensors. This data typically includes information about a user's gaze position or eye movements.

[0051] In a second step, the collected eye tracking data is preprocessed to remove noise and artifacts. This may involve filtering, calibration, and normalization techniques to ensure data quality.

[0052] Then, in a third step, the preprocessed eye tracking data is combined with corresponding input images or visual stimuli. These input images serve as the input to the CNN, while the eye tracking data acts as the ground truth or labels.

[0053] Next, in the fourth step, the CNN architecture is designed, which involves selecting appropriate layers, filter sizes, pooling operations, and activation functions. The architecture should be tailored to the given eye tracking task, taking into account factors such as input image resolution, eye movement patterns, and desired accuracy.

[0054] The CNN is trained using the generated dataset. The input images are fed into the network, and the predicted gaze position or eye movement is compared with the ground-truth labels. This is the layer in the CNN where cropping can be applied. ROIs (regions of interest) or POIs (points of interest) can be selected, with the other regions of the input data being cropped. The difference between the predicted and actual eye movements is used to calculate a loss function.

[0055] In the next, sixth step, backpropagation is performed to update the CNN's weights and biases based on the calculated loss. The gradient of the loss function with respect to the network parameters is calculated, and the weights are adjusted using optimization algorithms. The clipping mask can be approximated by a suitable function, such as those mentioned above.

[0056] The trained CNN is then evaluated on a separate validation dataset to assess its performance. Hyperparameters such as learning rate, batch size, and network architecture can be tuned to improve the model's accuracy and generalization ability.

[0057] After the CNN is trained and validated, it can be used for eye tracking predictions on new, unseen data. The input images are fed into the trained network, and the CNN generates predictions of gaze positions or eye movements. The CNN's output can be preprocessed and visualized to provide meaningful insights. This may involve mapping the predicted gaze positions to the input images or generating heatmaps to highlight regions of visual attention.

[0058] Finally, the performance of the eye-tracking CNN can be further improved through iterative cycles of training, validation, and fine-tuning. This may involve collecting more data, refining the network architecture, or incorporating additional techniques such as data augmentation or regularization. However, this step can be shortened due to the trainable clipping mask layer.

[0059] In another aspect, the invention relates to a computer program comprising program code for running a computer-implemented method as described above when the computer program is run on a computer.

[0060] In a further aspect, the invention relates to a computer-readable data carrier comprising program code of a computer program for running a computer-implemented method as described above when the computer program is run on a computer.

[0061] In a further aspect, the invention relates to a system for training a convolutional neural network using a clipping mask, the system comprising means for running a computer-implemented method as described above.

[0062] The described embodiments and further developments can be combined as desired.

[0063] Further possible embodiments, further developments and implementations of the invention also include combinations of features of the invention described above or below with reference to the embodiments that are not expressly mentioned. Short description of the drawings

[0064] The accompanying drawings are intended to provide a further understanding of embodiments of the invention. They illustrate embodiments and, together with the description, serve to explain principles and concepts of the invention.

[0065] Other embodiments and many of the mentioned advantages will become apparent with reference to the drawings. The elements shown in the drawings are not necessarily drawn to scale with respect to one another.

[0066] They show: Fig. 1 is a schematic view of a method according to an embodiment of the invention; Fig. 2 a schematic representation of the principle of the invention; Fig. 3 a sigmoid function as can be used for the invention; and Fig. 4 different functions as they can be used according to the invention.

[0067] In the figures of the drawings, identical reference numerals designate identical or functionally identical elements, parts or components, unless otherwise stated.

[0068] Fig. 1 shows a schematic view of a method according to an embodiment of the invention.

[0069] In a first step S10, training data is collected and preprocessed to remove, for example, noise and artifacts. This may involve filtering, calibration, and normalization techniques to ensure data quality. The preprocessed training data is labeled, which is necessary to find the ground truth or to identify correct processing of the data by the CNN.

[0070] In step S12, the CNN architecture is designed, which involves selecting appropriate layers, filter sizes, pooling operations, and activation functions. The CNN design also includes initializing the values, weights, and biases for each node in the CNN.

[0071] In step S14, the training data is passed through the CNN. The input images are fed into the network, and the network finds predictions for each training set. The prediction will be more or less accurate compared to the labels.

[0072] The difference between the predicted and actual labels is used to calculate a loss function in step S16. The loss function is the core element of the training process because it is used to refine the weights, biases, and values ​​for each node in the network.

[0073] In step S18, backpropagation is performed to update the weights and biases of the CNN based on the calculated loss. The gradient of the loss function with respect to the network parameters is calculated, and the weights are adjusted using optimization algorithms. The clipping mask can be approximated by a suitable function, such as a sigmoid function.

[0074] Steps S14 to S18 can be repeated until the loss function has a sufficiently low value.

[0075] The trained CNN is then evaluated on a separate validation dataset to assess its performance. Hyperparameters such as learning rate, batch size, and network architecture can be tuned to improve the model's accuracy and generalization ability.

[0076] Once the CNN is trained and validated, it can be used for original purposes on new, unseen data. The input data is fed into the trained network, and the CNN generates predictions according to its strategy. The CNN's output can be preprocessed and visualized to provide meaningful insights. This may involve mapping the output over relevant parameters or generating heatmaps to highlight areas of visual attention.

[0077] Finally, the performance of the CNN can be further improved through iterative cycles of training, validation, and fine-tuning. This may involve collecting more data, refining the network architecture, or incorporating additional techniques such as data augmentation or regularization. However, this step can be shortened by applying a clipping mask, which can be trained with the remaining layers of the CNN.

[0078] Fig. Figure 2 shows a schematic representation of a pruned CNN 10. The CNN may comprise multiple layers 12, each layer having multiple structures 14. Each structure 14 may be processed in each layer 12.

[0079] However, one of the layers 12 shown is a clipping mask 16. The structures 14 of the clipping mask 16 are arranged to multiply the output from a layer 18 before the clipping mask 16 by a clipping parameter M. Thus, the output of a structure s i with the clipping parameter M corresponding to structure 14 i multiplied.

[0080] The clipping parameter M i may be 1 to pass the output of the structure of layer 18 as input to a layer 20 beyond the clipping mask. Otherwise, the clipping parameter may be 0 to clip a structure 18. In Fig. 2, four structures 14 are shown as being clipped, while another four structures 14 are shown as passing the structure output of layer 18 as the structure input to layer 20.

[0081] Fig. 3 shows the graph of a sigmoid function. The function shown is sig(t)=11+e−t

[0082] When rounded, the function (sig(t)) returns values ​​of 1 for any t greater than or equal to 0 and values ​​of 0 for any t less than 0. When t is substituted with the helper vector Z, there can be values ​​of 1 or 0 depending on the helper vector Z.

[0083] Fig. Figure 4 shows two functions for f(Z), where Z is the helper vector. One of the functions f(Z) is the rounding function round(Z), which returns values ​​of 1 for positive Z and 0 for negative Z. This function can be used to determine a pruning parameter, for which Z is the corresponding parameter to be trained in the CNN.

[0084] The rounding function doesn't need a gradient because it's not a continuous function. It changes from 0 to 1 when Z equals 0, resulting in a differential of infinity. However, to train Z, a gradient must be derived to determine whether Z needs to be incremented or decremented.

[0085] The gradient function for the rounding function is approximated by an approximation function which roundapprox(Z′)=Z′−sin(2πZ′)2π can be. As in Fig. 4 shown is round approx (Z') is a continuous function from which a gradient can be derived.

[0086] The combination of both functions enables a trainable clipping mask for a CNN. One function is used for forward propagation, and the other can be applied during backpropagation. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] He et al., Channel pruning for accelerating very deep neural networks, CoRR, abs / 1707.06168, 2017

[0008] Gao et al., Discrete model compression with resource constraint for deep neural networks, IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1896-1905, 2020

[0009] Gao et al., Disentangled differentiable network pruning, In European Conference on Computer Vision, 2022

[0010]

Claims

[1] Computer-implemented method for training a convolutional neural network comprising nodes arranged in layers and a clipping mask, the method comprising the following steps: - Providing at least one set of labelled training data (S10); - initializing the convolutional neural network (S12) comprising at least one convolutional layer and a clipping mask behind the convolutional layer; - Passing the training data through the convolutional neural network (S14); - Calculating a loss function (S16) and comparing the loss function with labels of the training data to quantify the convolutional neural network process; and - Minimizing the loss function by backpropagation (S18), including determining a gradient of the loss function to determine a direction for optimizing the nodes of the convolutional neural network; where passing the training data through the clipping mask multiplying the structures s i entering the clipping mask with a clipping parameter M i where the clipping parameter M i for the s i the structure is 0 or 1; where the clipping mask is approximated by an approximation function during backpropagation. [2] A computer-implemented method according to claim 1, wherein the clipping parameter M i from a helper vector Z i is determined, where the helper vector Z i is a trainable parameter of the convolutional neural network and where the approximation function is a function of Z i is. [3] A computer-implemented method according to claim 2, wherein the clipping parameter is derived from rounding a sigmoid function, wherein the helper vector Z i is the input of the sigmoid function. [4] A computer-implemented method according to claim 3, wherein the helper vector Z i is initialized with a value of 0 or close to 0. [5] Computer-implemented method according to one of the preceding claims, wherein the sum of all clipping parameters M i is greater than 0. [6] Computer-implemented method according to one of the preceding claims, wherein the sum of all clipping parameters M i less than or equal to the number of output structures of the convolutional layer s out , divided by the compression ratio of the clipping mask, or where the sum of all clipping parameters M i is less than or equal to a factor corresponding to the hardware limitations of the target operating system. [7] Computer-implemented method according to one of the preceding claims, wherein the convolutional neural network is designed for processing image or audio data and is implemented in particular as an eye tracking application. [8] A computer program comprising program code for running a computer-implemented method according to any one of the preceding claims when the computer program is run on a computer. [9] A computer-readable medium comprising program code of a computer program for running a computer-implemented method according to any one of claims 1 to 7 when the computer program is run on a computer. [10] A system for training a convolutional neural network using a clipping mask, the system comprising means for running a computer-implemented method according to any one of claims 1 to 7.

Images