Method and device for managing the training of an artificial neural network

Abstract

The invention relates to a method for managing the training of a neural network. The invention also relates to a computer device (100) implementing such a method. Figure for the abstract: 1

Description

Title of the invention: Method and device for managing the training of an artificial neural network Technical field of the invention

[0001] The present invention relates to the field of machine learning. In particular, the invention relates to a method for managing the training of a neural network for performing a classification task. The invention also relates to a computer device implementing such a method. Prior art

[0002] The standard approach for training a neural network for classification involves using softmax loss, which consists of a softmax layer and cross-entropy loss. However, it is widely observed that this loss only helps the network to produce separable, but not sufficiently discriminating, features. In many problems, intraclass variation is very high, meaning that the samples within each class are highly diverse. At the same time, interclass separability is low, meaning that there are samples from different but very similar classes. This makes prediction much more difficult. Therefore, to achieve optimal generalization capabilities, a good machine learning tool in general, and a neural network in particular, must learn to produce features with high intraclass compactness and high interclass separability.

[0003] In some prior work, a center loss is applied in parallel with the standard softmax loss. This loss encourages directly minimizing the distance from each point to its corresponding class centroid (in feature space). However, this can lead to feature collapse, where all samples of each class reduce to a single point.

[0004] In other previous work, the advantages of a large margin in the context of deep learning are highlighted. However, this work ignores the bias terms of the softmax layer, and consequently, the margin can be viewed via angles between vectors instead of the Euclidean distance. Furthermore, this work only encourages the interclass margin and does not take intraclass compactness into account. Yet, in scenarios where variance is significant, taking intraclass compactness into account proves necessary. Summary of the invention

[0005] The invention aims to remedy these shortcomings. Its purpose is to provide an alternative solution for training a neural network that can be applied to any neural network deep learning tool.

[0006] To achieve this goal, the invention relates, according to a first aspect, to a method for managing, by means of a computer device, the training of a neural network, the method comprising the steps of: i. determine a first loss value by means of a discriminant loss function applied to the penultimate layer of said network, said discriminant loss function comprising a compactness term, a separability term and a regularization term; ii. determine a second loss value by means of a softmax loss function applied to the last layer of said network; iii. determine a third loss value based on the first loss value and the second loss value.

[0007] According to one variant, said compactness term can be established as a function of the distance L2 between a sample of a class and the centroid of the class and as a function of a first predefined parameter.

[0008] According to another variant, said separability term can be determined as a function of a maximization of the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter.

[0009] According to yet another variant, said regularization term can be obtained by determining the centroid of a class on the basis of a sample or as a function of a variation between an estimate of the centroid at a first instant and an estimate of the centroid at a second instant.

[0010] According to a second aspect, the invention relates to a device for managing the training of a neural network, the device comprising at least one information processing unit, including at least one processor, and a data storage medium configured to implement a method as described above.

[0011] According to a third aspect, the invention relates to a computer program comprising program code instructions for the execution of the steps of a process as described above when said program is executed by at least one processor.

[0012] According to a fourth aspect, the invention relates to a medium usable in a computer on which a program as described above is recorded. Brief description of the figures

[0013] Other features and advantages of the invention will become apparent from an examination of the detailed description below, and the accompanying figures, in which:

[0014] [Fig-1] is a diagram illustrating the implementation of a process according to the invention;

[0015] [Fig.2] is a functional diagram of a device according to the invention;

[0016] [Fig.3] is a flowchart of the steps of a process according to the invention;

[0017] [Fig.4] illustrates experimental results of the implementation of the process according to the invention;

[0018] [Fig. 5] illustrates experimental results of the implementation of the process according to the invention; and

[0019] [Fig.6] illustrates experimental results of the implementation of the process according to the invention. Detailed description of the invention

[0020] Figure 1 illustrates the implementation of the method for managing the training of a neural network according to the invention. According to the invention, a discriminant loss function is applied to the penultimate layer of the network, just before the softmax layer. Since this discriminant loss function is applied only to this layer, it can advantageously be used with any deep neural network for end-to-end training. As will be seen, this discriminant loss function combines two distinct terms: a first term that estimates an articulated center loss and a second term that estimates a margin loss.

[0021] The loss of articulated center is responsible for compactness; by minimizing this loss, the intraclass compactness of the resulting features is maximized. In particular, it encourages the distance from each point to its centroid to be smaller than a predefined positive parameter ôv. This helps to avoid the collapse phenomenon.

[0022] On the other hand, margin loss encourages maximizing the class margin. The class margin of a given class is defined as the smallest distance between the samples of that class and its nearest decision boundary. To this end, one first derives an exact analytical formula for the decision boundaries. Then, instead of maximizing the margin of each training sample with respect to the decision boundaries, one calculates the exact analytical formula for the centroid of the class with respect to the decision boundary and constrains it to be greater than a predefined parameter ôd.

[0023] Advantageously, such a discriminant loss function simultaneously models the compactness of the class and the margin relative to the decision boundaries in a softmax network, by means of an explicit formula, without any approximation. Applied in parallel with the softmax loss, it strengthens the discriminating power of the features. Compactness is ensured by imposing that the The distance between each point and its corresponding centroid must be less than a first predefined parameter, ôv. Separability is ensured by requiring that the distance between each centroid and the decision boundaries be greater than a second predefined parameter, ôd. By applying this loss to the penultimate layer, the decision boundaries can be explicitly calculated (they are hyperplanes), and consequently, the distance to these boundaries can also be explicitly calculated. Finally, the loss thus determined is combined with the standard softmax loss to obtain the total loss for managing network training.

[0024] Conventionally, the implementation of the method according to the invention described above is carried out by means of a device 100 for managing the training of a neural network as defined in the present invention, which is schematically illustrated in [Fig. 2]. It is essentially a computer device, which includes at least one information processing unit 101, comprising one or more processors, a data storage medium 102, for example rewritable volatile memory or rewritable non-volatile memory, on which is stored, in particular, a program which includes program code instructions for the execution of the steps of the method according to the invention described below, and an input and output interface 103 enabling the reception and transmission of data.The device 100 according to the invention is, for example, a computer, a server, a supercomputer, or a combination of one or more of these devices, possibly interconnected by means of a communication network.

[0025] According to the invention, the elements described above contribute to enabling the implementation of a method for managing the training of a neural network, as described below in relation to [Fig.3] which illustrates by means of a flowchart the steps of the method according to the invention.

[0026] According to a first step 301 of the method according to the invention, the device 100 according to the invention determines a first loss value by means of a discriminant loss function applied to the penultimate layer of the network, this discriminant loss function comprising a compactness term, a separability term and a regularization term. Preferably, the compactness term is established as a function of the distance L2 between a sample of a class and the centroid of the class and as a function of a first predefined parameter (ôj). Furthermore, the separability term is preferably determined as a function of a maximization of the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter (ôv).Furthermore, the regularization term is preferably obtained by determining the centroid of a class on the basis of a sample or as a function of a variation between an estimate of the centroid of a class and a first. instant and an estimation of the centroid of a class at a second instant. Then, according to a second step 302 of the method according to the invention, the device 100 according to the invention determines a second loss value by means of a softmax loss function applied to the last layer of the network. Finally, according to a third step 303 of the method according to the invention, the device 100 according to the invention determines a third loss value as a function of the first loss value and the second loss value.

[0027] For greater clarity, we will explain below the basics which establish the context of the invention.

[0028] Consider a classification problem with C classes and an input space X (which could be images, time series, vectors, etc.). The neural network transforms the inputs into fixed-dimensional vectors: f0 : X —> F → R d, where 0 is the set of parameters of the neural network and d is the dimension of the feature space F. For example, if we consider an input q e X, its corresponding feature in the feature space F is x := f0(q) e F. It then passes through a softmax layer consisting of a linear transformation (Eq. (1)) and a softmax function (Eq. (2)):

[0029] [Math.Eq(l)] z = Wx + b, and

[0030] [Math.Eq(2)]

[0031] where Z is the i-th component of the column vector z. By denoting g(x) = o(W x + b), the prediction is then made by the class whose value is maximal, i.e., y'' = arg maxi o(z)i = arg maxi zi. In the standard approach, to encourage the neural network to predict the maximum score for the correct class, the softmax loss function is adopted, i.e.:

[0032] [Math.Eq(3)]

[0033] where B is the mini-batch under consideration, q is the training example with its corresponding truth label ye {1, 2...C}. By minimizing this loss function with respect to 0 and (W, b), the network learns to assign the maximum score to the correct class.

[0034] We will now detail the discriminating loss function which, according to the invention, is advantageously applied to the penultimate layer of the neural network.

[0035] Considering the pair of classes {i, j}, we have:

[0036] [Math.Eq(4)] 7.^= (Wx+b).-(Wx+b)~ (W^x) + (b r bj)

[0037] where zi is the ith component of the column vector z, Wi is the ith row of the matrix W. Let Pij be the hyperplane defined by:

[0038] [Math.Eq(5)] (^,-^) + ( / ,,-^)=0

[0039] We then see that zi > zj and zi < zj correspond to the two half-spaces divided by the hyperplane Pij. Consequently, the decision boundary of the pair of classes {i, j} is the hyperplane Pij. For all points located on Pij, the scores assigned to the two classes are identical (since zi = zj).

[0040] According to equation (3), for an input of class i, we see that the softmax loss encourages zi > zj, Vj = i, that is<Wi, x> + bi ><Wj , x> + bj, that is to say<Wi - Wj , x> + (bi - bj ) > 0.

[0041] Consequently, the softmax loss forces features to lie on the correct side of the decision hyperplanes and is therefore suitable for encouraging linearly separable features. Furthermore, this loss encourages inputs of the same class to produce probability vectors that are close to each other and close to the corresponding corner of the unit simplex. However, we note a property of the softmax function:

[0042] [Math.Eq(6)] o(z) = o(z + &), VeeK

[0043] where z + e denotes the vector such that (z + e)i = zi + e. Consequently, even for two inputs of the same class that produce exactly the same probability vectors, it is not guaranteed that features of the same class will be close to each other. If this is the case, it is an intrinsic property of neural networks intelligent enough to do so, and not a consequence of using a "softmax" type loss. As already discussed above, intraclass compactness is important for improving the generalization capacity of the network. It is therefore desirable to have a loss that explicitly encourages this property for the produced features. This is why the discriminant loss function according to the invention is proposed as explained below.

[0044] To obtain a better classification, we rely on two factors: intraclass compactness and interclass separability. To obtain these properties, we work in the feature space F. In many classification problems, intraclass variance is very important because the samples of the same class are very diverse. Thus, by forcing the network to map the various samples of the same class into representations with intraclass compactness, the network learns the representative features of each class and ignores the unnecessary details (so that their representations are similar). On the other hand, there are samples belonging to different but very similar classes, which leads to poor classification. Therefore, we also want to learn a representation with a large margin between the classes. In this way, the network learns which characteristics differentiate the classes, thus enabling better classification of the samples. To achieve all these objectives, the discriminant loss function according to the invention advantageously takes the following form:

[0045] + P ' Lmarg + Y ' Lrf>g

[0046] where Lcomp, Lmarg and Lreg are respectively responsible for class compactness, inter-class separability and regularization. These three terms are now examined in detail.

[0047] Consider the current mini-batch B. We denote CB as the set of classes present in B, and CBc as the examples of class c in B. To obtain better intra-class compactness, we apply:

[0048] [Math.Eq(7)] Lcomp = X [ H m c - X || - O r ] J

[0049] where: II. It is the distance L2 and [x]+ = max(0, x). mc is the centroid of class c. This The function is only activated if `lime - xll` > 0v and is deactivated when `lime - xll` < 0v, i.e., when the distance from a point to its corresponding centroid is less than 0v. The goal of this function is therefore to ensure that the distance from each point to its centroid is less than 0v. One advantage of this function is that it only penalizes distances smaller than 0v, whereas other center loss methods directly penalize the distance from each point to its centroid, thus encouraging all samples to collapse to a single point.

[0050] To obtain better inter-class separability, we want a loss function that imposes a large margin between classes while taking decision boundaries into account. A simple strategy would be to maximize the distance of each sample to all decision boundaries. However, this method is very expensive. Instead, we maximize the distance of each centroid to the decision boundaries with respect to all other classes. The margin loss is thus represented as follows:

[0051] [Math.Eq(8)] ^ mrg = ôèî x max( [ - d(P) ci ) sign ( g ( m c ) ) - g ( m c 1)

[0052] Thus, when the centroid mc is on the correct side of the decision boundary, sign(g(mc)c - g(mc)i) > 0, and we therefore minimize [ôd - d(mc,Pci)]+, consequently d(mc,Pci) is encouraged to be greater than ôd. Conversely, if mc is on the wrong side of the At the decision boundary, we minimize [ôd + d(mc,Pci)]+. This forces me to move to the correct side.

[0053] Therefore, this loss is only deactivated if the centroid is on the correct side with respect to the decision boundaries and its distance from the decision boundaries is greater than ôd. Furthermore, the aggregation operation maxi=c is chosen instead of meani=c. The reason for this is that some class pairs are easier to separate than others. Therefore, if mean aggregation is used, simply focusing on the easy pairs and ignoring the difficult ones, the overall loss is always minimized. On the contrary, with max aggregation, the neural network is forced to focus on the difficult pairs so that it can learn more useful features to increase discriminatory power. Note that the distance from me to the hyperplane Pci (which is the decision boundary of the class pair (c, i)) can be explicitly calculated as follows:

[0054] [Math.Eq(9)] d(mc,Pü)-

[0055] Thus, at this stage, the discriminant loss function according to the invention encourages each centroid to move away from the decision boundaries. However, there is no guarantee that the decision boundaries form a near space, and the resulting centroids could be pushed to infinity. To solve this problem, the discriminant loss function according to the invention contains a regularization term, which takes the following form:

[0056] [Math.Eq(lO)] i „„Ly iimi|

[0057] Furthermore, the centroid of a class can be calculated in several ways. One way is to use a sample of the elements of a class. From a mathematical point of view, this amounts to determining a centroid of a class by solving:

[0058] [Math.Eq(ll)] mtc ~ =

[0059] Another way is to determine the difference between an estimate of the centroid at a first time and an estimate of the centroid at a second time. From a mathematical point of view, this amounts to solving:

[0060] [Math.Eq(12)] ~ m^ariation = y ■ + ( 1 - y ) • mFcurrent

[0061] Figure 4 illustrates experimental results obtained on the CIFAR10 dataset. First, it is evaluated how the discriminant loss function according to the invention improves class compactness and inter-class separability. compared to softmax loss. To do this, we first train the network on the CIFAR10 training set. Then, we calculate the characteristics of the test set. Finally, we use the t-SNE technique (Van der Maaten and Hinton 2008) to visualize the characteristics in a two-dimensional space. As seen in the upper left of the figure, softmax loss does not allow for proper separation of classes in the feature space. Conversely, as seen in the upper right of the figure, the discriminant loss function according to the invention allows for better inter-class separability.

[0062] Next, we evaluate how the discriminant loss function according to the invention helps the network to generalize better to the test set by examining the test accuracy. We use only a portion of the training set for training, and then evaluate on the complete test set, with the softmax loss and the discriminant loss function according to the invention. The results are presented in the table reproduced in the lower part of the figure, with the softmax loss (Ls) results in the second row of the table and the results of the discriminant loss function according to the invention in the third row of the table (Ldiscr). We observe that at different percentages of the training set, the method according to the invention consistently outperforms the softmax loss.Furthermore, with only 60% of the training set, the method according to the invention already outperforms the softmax loss using the full training set. This result proves that the discriminant loss function according to the invention helps the network to generalize better using less training data.

[0063] Figure 5 illustrates the same experiments performed on the SVHN dataset. As can be seen in the figure, comparing the results of the softmax loss method on the left with those of the discriminant loss function according to the invention on the right, improved separation and compactness are obtained. Furthermore, as shown in the table illustrated in the lower part of the figure, the method according to the invention consistently outperforms the softmax loss method. Remarkably, with only 40% of the training dataset, the method according to the invention already achieves the same test accuracy as the softmax network using the complete training dataset. This result proves that the discriminant loss function helps the network to generalize more effectively using less training data.

[0064] As already mentioned, the discriminant loss function according to the invention is applied to the penultimate layer of the network (just before the softmax layer). Neural networks are generally made up of layers that can be grouped into two subsets. A first subset of layers dedicated to feature extraction, and a second sub- The second subset, dedicated to classification, comprises the last layer(s) of the neural network. It is this second subset, in particular, that is referred to when discussing the softmax layer. It is therefore interesting to observe the impact of this loss function on the overall network, as the results presented in Figures 4 and 5 correspond precisely to the characteristic of the penultimate layer. Consequently, it can be assumed that the discriminant loss function according to the invention only impacts the fully connected layers that provide discriminating features and that it has little or no impact on the feature extractor. This is why the same t-SNE technique is applied to the intermediate layers of the feature extractor. In the two experiments above, ResNetl8 (He et al. 2016) was used as the feature extractor, followed by several fully connected layers before the softmax layer.ResNetl8 comprises four main blocks, each yielding outputs of dimension H x W x Ci, where H and W are spatial dimensions (and therefore depend on the input dimension), and Ci is the number of channels in block i (independent of the input dimension). First, global means are pooled along the spatial dimensions to obtain a feature vector of dimension Ci (for each block i). Then, t-SNE is performed as in the penultimate layer. The results for CIFAR10 and SVHN are shown in [Fig. 6], with the CIFAR10 dataset at the top and the SVHN dataset at the bottom.

[0065] For both datasets, we observe that there are no distinct clusters in the first two blocks. This is normal because these are shallow layers and no semantic features are actually captured. However, starting with block 3, the discriminant loss function according to the invention appears to help the network produce more distinctive clusters compared to softmax loss. This effect is also observed for block 4 in both datasets. This implies that the discriminant loss function according to the invention not only impacts the features of the penultimate layer, but actually helps the network learn more discriminating features at intermediate levels. In conclusion, the discriminant loss function according to the invention is useful for network training.

[0066] Therefore, thanks to the method and device according to the invention described above, a solution is provided for training a neural network that can be applied to any neural network deep learning system. A neural network thus trained can, in particular, be used for various classification tasks, for example, but not limited to, facial recognition, traffic sign identification, and behavioral recognition. driver behavior of a vehicle, identification of road surface type, voice recognition, etc.

Claims

Demands

1. A method for managing, by a computer device (100), the training of a neural network, characterized in that said method comprises the steps of: i. determining a first loss value by means of a discriminant loss function applied to the penultimate layer of said network, said discriminant loss function comprising a compactness term, a separability term and a regularization term; ii. determining a second loss value by means of a softmax loss function applied to the last layer of said network; iii. determining a third loss value as a function of the first loss value and the second loss value.

2. Method according to claim 1, characterized in that said compactness term is established as a function of the distance L2 between a sample of a class and the centroid of the class and as a function of a first predefined parameter.

3. A method according to any one of the preceding claims, characterized in that said separability term is determined as a function of a maximization of the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter.

4. A method according to any one of the preceding claims, characterized in that said regularization term is obtained by determining the centroid of a class on the basis of a sample or as a function of a variation between an estimate of the centroid of a class at a first instant and an estimate of the centroid of a class at a second instant.

5. Device (100) for managing the training of a neural network, characterized in that the device comprises at least one information processing unit (101), comprising at least one processor, and a data storage medium (102) configured to implement a method according to any one of the preceding claims.

6. A computer program comprising program code instructions for carrying out the steps of a process according to one

7. any of claims 1 to 4 when said program is executed by at least one processor. A storage medium usable in a computer, characterized in that a program according to claim 6 is stored thereon.

Images