Method and device for managing training of an artificial neural network using a discriminative loss function and a softmax loss function

Abstract

The invention relates to a method for managing the training of a neural network. The invention also relates to a computer system (100) implementing such a method.

Description

DESCRIPTION METHOD AND DEVICE FOR MANAGING THE TRAINING OF AN ARTIFICIAL NEURAL NETWORK IN 111RE: USING A DISCRIMINATING LOSS FUNCTION AND A SOFTMAX LOSS FUNCTION Technical field of the invention

[0001] The present invention relates to the field of machine learning. In particular, the invention relates to a method for managing the training of a neural network for performing a classification task. The invention also relates to a computer system implementing such a method. Prior art

[0002] The standard approach for training a neural network for classification involves using softmax loss, which consists of a softmax layer and cross-entropy loss. However, it is widely observed that this loss only helps the network produce separable features, not sufficiently discriminating ones. In many problems, intraclass variation is very high, meaning that the samples within each class are highly diverse. At the same time, interclass separability is low; that is, there are samples from different but very similar classes. This makes prediction much more difficult. Therefore, to achieve optimal generalization capabilities, a good machine learning tool in general, and a neural network in particular, must learn to produce features with high intraclass compactness and high interclass separability.

[0003] In some earlier work, a center loss is applied in parallel with the standard softmax loss. This loss encourages directly minimizing the distance from each point to its corresponding class centroid (in feature space). However, this can lead to a feature collapse, where all samples of each class reduce to a single point.

[0004] Previous work has highlighted the advantages of a large margin in the context of deep learning. However, this work ignores the bias terms of the softmax layer, and consequently, the margin can be viewed through angles between vectors instead of Euclidean distance. Furthermore, this work only encourages the interclass margin and does not consider intraclass compactness. Yet, in scenarios where variance is significant, taking intraclass compactness into account becomes necessary. Summary of the invention

[0005] The invention aims to overcome these shortcomings. Its purpose is to provide an alternative solution for training a neural network that can be applied to any neural network deep learning tool.

[0006] To achieve this goal, the invention relates, according to a first aspect, to a method of managing, by a computer device, the training of a neural network, the method comprising the steps of: i) determining a first loss value by means of a discriminant loss function applied to the penultimate layer of said network, said discriminant loss function comprising a compactness term, a separability term and a regularization term; ii) determining a second loss value by means of a softmax loss function applied to the last layer of said network; iii) determining a third loss value as a function of the first loss value and the second loss value.

[0007] It is clear from this text that the term "compactness" is used to ensure the compactness of each class. As is well known, the compactness of a class is represented by the maximum distance between two points out of all the points in said class.

[0008] It is clear from this text that the term "separability" is used to ensure the separability of each class. As is well known, the separability of a class is represented by the minimum distance between a point in that class and the decision boundaries of that class.

[0009] It is clear from this text that the regularization term is configured to prevent the centroids of the classes from being pushed to infinity. It is represented by the vector norms measured between each point of each class and the origin of a coordinate system in which these points are defined. Since training a neural network involves minimizing the loss function(s), the regularization term will minimize the values ​​taken by these vector norms.

[0010] It is clear from this text that the third loss value results from the combination of the first and second loss values. Advantageously, the third loss value is a simple linear combination of the first and second loss values. In an extreme case, the weighting coefficient assigned to the second loss value becomes zero.

[0011] According to one variant, the compactness term can be established as a function of the distance L2 between a sample of a class and the centroid of the class, and as a function of a first predefined parameter. Preferably, the compactness term is adapted to impose that the distance L2 between each sample of a class and the centroid of the class be less than a first predefined parameter (δv).

[0012] According to another variant, the separability term can be determined based on a maximization of the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter. Preferably, said separability term is adapted to impose that a distance between each centroid of a class and the decision boundaries of said class is greater than a second predefined parameter (ôd).

[0013] According to yet another variant, the said regularization term can be obtained by determining the centroid of a class on the basis of a sample or as a function of a variation between an estimate of the centroid at a first instant and an estimate of the centroid at a second instant.

[0014] According to a second aspect, the invention relates to a device for managing the training of a neural network, the device comprising at least one information processing unit, including at least one processor, and a data storage medium configured to implement a method as described above.

[0015] According to a third aspect, the invention relates to a computer program comprising program code instructions for executing the steps of a process as described above when said program is executed by at least one processor.

[0016] According to a fourth aspect, the invention relates to a medium usable in a computer on which a program as described above is recorded. Brief description of the figures

[0017] Other features and advantages of the invention will become apparent from the detailed description below and the accompanying figures, in which:

[0018] [Fig. 1] is a diagram illustrating the implementation of a process according to the invention;

[0019] [Fig. 2] is a functional diagram of a device according to the invention;

[0020] [Fig. 3] is a flowchart of the steps of a process according to the invention;

[0021] [Fig. 4] illustrates experimental results of the implementation of the process according to the invention;

[0022] [Fig. 5] illustrates experimental results of the implementation of the process according to the invention; and

[0023] [Fig. 6] illustrates experimental results of the implementation of the process according to the invention. Detailed description of the invention

[0024] Figure 1 illustrates the implementation of the neural network training management method according to the invention. According to the invention, a discriminant loss function is applied to the penultimate layer of the network, just before the softmax layer. Since this discriminant loss function is applied only to this layer, it can advantageously be used with any deep neural network for end-to-end training. As will be seen, this discriminant loss function combines two distinct terms: a first term that estimates a center-articulated loss and a second term that estimates a margin loss.

[0025] The loss of articulated center is responsible for compactness; by minimizing this loss, the intraclass compactness of the resulting features is maximized. In particular, it encourages the distance from each point to its centroid to be smaller than a predefined positive parameter θ. vThis helps to prevent the collapse phenomenon.

[0026] On the other hand, margin loss encourages maximizing the class margin. The class margin of a given class is defined as the smallest distance between the samples of that class and its nearest decision boundary. To this end, we first derive an exact analytical formula for the decision boundaries. Then, instead of maximizing the margin of each training sample with respect to the decision boundaries, we calculate the exact analytical formula of the centroid of the class with respect to the decision boundary and it is forced to be greater than a predefined parameter ôd.

[0027] Advantageously, such a discriminant loss function simultaneously models the class compactness and the margin from decision boundaries in a softmax network, using an explicit formula without any approximation. Applied in parallel with the softmax loss, it enhances the discriminating power of the features. Compactness is ensured by requiring that the distance between each point and its corresponding centroid be less than a first predefined parameter θ vSeparability is ensured by requiring that the distance between each centroid and the decision boundaries be greater than a second predefined parameter, Δd. By applying this loss to the penultimate layer, the decision boundaries can be explicitly calculated (they are hyperplanes), and consequently, the distance to these boundaries can also be explicitly calculated. Finally, the loss thus determined is combined with the standard softmax loss to obtain the total loss for managing the network training (see the "+" symbol in Figure 1, representing a sum such as a linear combination).

[0028] Conventionally, the implementation of the method according to the invention described above is carried out by means of a device 100 for managing the training of a neural network as defined in the present invention, which is schematically illustrated in Figure 2. This device is essentially a computer system, comprising at least one information processing unit 101, including one or more processors, a data storage medium 102, for example, rewritable volatile memory or rewritable non-volatile memory, on which is stored, in particular, a program comprising program code instructions for executing the steps of the method according to the invention described below, and an input / output interface 103 for receiving and transmitting data. The device 100 according to the invention is, for example, a computer, a server, a supercomputer, or a combination thereof. one or more of these devices, possibly interconnected by means of a communication network.

[0029] According to the invention, the elements described above contribute to enabling the implementation of a method for managing the training of a neural network, as described below in relation to Figure 3, which illustrates by means of a flowchart the steps of the method according to the invention.

[0030] According to a first step 301 of the method according to the invention, the device 100 according to the invention determines a first loss value by means of a discriminant loss function applied to the penultimate layer of the network, this discriminant loss function comprising a compactness term, a separability term, and a regularization term. Preferably, the compactness term is established as a function of the distance L2 between a sample of a class and the centroid of the class and as a function of a first predefined parameter (8v). Furthermore, the separability term is preferably determined as a function of maximizing the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter (ôd).Furthermore, the regularization term is preferably obtained by determining the centroid of a class based on a sample or as a function of the variation between an estimate of the centroid of a class at a first time and an estimate of the centroid of a class at a second time. Then, according to a second step 302 of the method according to the invention, the device 100 according to the invention determines a second loss value by means of a softmax loss function applied to the last layer of the network. Finally, according to a third step 303 of the method according to the invention, the device 100 according to the invention determines a third loss value as a function of the first loss value and the second loss value.

[0031] For greater clarity, we will explain below the basics that establish the context of the invention.

[0032] Consider a classification problem with C classes and an input space X (which could be images, time series, vectors, etc.). The neural network transforms the inputs into fixed-dimensional vectors: f9 : X → F ∈ R d, where 0 is the set of parameters of the neural network and d is the dimension of the feature space F. For example, if we consider an input q ∈ EX, its corresponding feature in the feature space F is x := f9(q) ∈ F. It then passes through a softmax layer consisting of a linear transformation (Eq. (1)) and a softmax function (Eq. (2)):

[0033] [Math.Eq(l)] z = Wx + b, WER Cxd and b ER c

[0034] [Math.Eq(2)]

[0035] where Zi is the i-th component of the column vector z. Denoting g(x) = o(W x + b), the prediction is then made by the class whose value is maximal, that is, y" = arg maxi o(z)i = arg maxi zi. In the standard approach, to encourage the neural network to predict the maximum score for the correct class, the softmax loss function is adopted, i.e.:

[0036] [Math.Eq(3)]

[0037] where B is the mini-batch under consideration, q is the training example with its corresponding truth label y E {1, 2...C}. By minimizing this loss function with respect to 0 and (W, b), the network learns to assign the maximum score to the correct class.

[0038] We will now detail the discriminant loss function which, according to the invention, is advantageously applied to the penultimate layer of the neural network.

[0039] Considering the pair of classes {i, j}, we have:

[0040] [Math.Eq(4)]

[0041] where zi is the ith component of the column vector z, and Wi is the ith row of the matrix W. Let Pij be the hyperplane defined by:

[0042] [Math.Eq(5)] {W i - W j ,x) + b i - b j ) = G

[0043] We then see that zi > zj and zi < zj correspond to the two half-spaces divided by the hyperplane Pij. Consequently, the decision boundary of the pair of classes {i, j} is the hyperplane Pij. For all points located on Pij, the scores assigned to the two classes are identical (since zi = zj).

[0044] According to equation (3), for an input of class i, we see that the softmax loss encourages zi > zj, Vj / = i, that is<Wi , x> + bi ><Wj , x> + bj, that is to say<Wi - Wj , x> + (bi - bj ) > 0.

[0045] Therefore, the softmax loss forces features to lie on the correct side of the decision hyperplanes and is thus suitable for encouraging linearly separable features. Furthermore, this loss encourages inputs of the same class to produce probability vectors that are close to each other and close to the corresponding corner of the unit simplex. However, we note a property of the softmax function:

[0046] [Math.Eq(6)] o(z) = o(z + E), Vs GR

[0047] where z + e denotes the vector such that (z + c)i = zi + E. Therefore, even for two inputs of the same class that produce exactly the same probability vectors, it is not guaranteed that features of the same class will be close to each other. If this is the case, it is an intrinsic property of neural networks intelligent enough to do so, and not a consequence of using a "softmax" type loss. As already discussed above, intraclass compactness is important for improving the network's generalization capacity. It is therefore desirable to have a loss that explicitly encourages this property for the produced features. This is why the discriminant loss function according to the invention, as explained below, is proposed.

[0048] To obtain a better classification, we rely on two factors: within-class compactness and between-class separability. To obtain these properties, we work in the feature space F. In many classification problems, within-class variance is very important because samples within the same class are highly diverse. Thus, by requiring the network to map the various samples of the same class into representations with within-class compactness, the network learns the representative features of each class and ignores irrelevant details (so their representations are similar). On the other hand, there are samples belonging to different but very similar classes, which leads to poor classification. This is why we also want to learn a representation with a large margin between classes.In this way, the network learns which characteristics differentiate the classes, allowing for better classification of the samples. To achieve all of these... For objectives, the discriminating loss function according to the invention advantageously takes the following form:

[0050] where Lcomp, Lmarg and Lreg are respectively responsible for class compactness, inter-class separability and regularization. We now examine these three terms in detail.

[0051] Consider the current mini-batch B. We denote CB as the set of classes present in B, C B c as in the examples of class c in B. To obtain better intraclass compactness, we apply:

[0052] [Math.Eq(7)]

[0053] where: | | . | | is the distance L2 and [x]+ = max(0, x). m cis the centroid of class c. This function is only activated if ||mc - x|| > 0v and vanishes when ||mc - x|| < 0v, that is, when the distance from a point to its corresponding centroid is less than 0v. The goal of this function is therefore to ensure that the distance from each point to its centroid is less than 0v. One advantage of this function is that it only penalizes distances smaller than 0v, whereas other center loss methods directly penalize the distance from each point to its centroid, thus encouraging all samples to collapse to a single point.

[0054] To achieve better inter-class separability, we want a loss function that imposes a large margin between classes while still accounting for decision boundaries. A simple strategy would be to maximize the distance of each sample to all decision boundaries. However, this method is very expensive. Instead, we maximize the distance of each centroid at the decision boundaries relative to all other classes. The margin loss is thus represented as follows:

[0055] [Math.Eq(8)] sign(^(m c ) c ) - ^(m c ) + )

[0056] Thus, when the centroid me is on the correct side of the decision boundary, sign(g(mc)c - g(mc)i) > 0, and we therefore minimize [ôd - d(mc,Pci)]+, consequently d(mc,Pci) is encouraged to be greater than ôd. Conversely, if me is on the wrong side of the decision boundary, we minimize [ôd + d(mc,Pci)]+. This forces me to move to the correct side.

[0057] Therefore, this loss is only disabled if the centroid is on the correct side of the decision boundaries and its distance from the decision boundaries is greater than Δd. Furthermore, we opt for the maximum aggregation operation instead of meani = c. The reason for this is that some class pairs are easier to separate than others. Consequently, if we use mean aggregation, simply focusing on the easy pairs and ignoring the difficult ones, the overall loss is always minimized. Conversely, with maximum aggregation, we force the neural network to focus on the difficult pairs so that it can learn more useful features to increase its discriminatory power. Note that the distance from me to the hyperplane Pci (which is the decision boundary of the class pair (c, i)) can be explicitly calculated as follows:

[0058] [Math.Eq(9)]

[0059] Thus, at this stage, the discriminant loss function according to the invention encourages each centroid to move away from the decision boundaries. However, there is no guarantee that the decision boundaries form a near space, and the resulting centroids could be pushed to infinity. To solve this problem, the discriminant loss function according to the invention contains a regularization term, which takes the following form:

[0060] [Math.Eq(lO)]

[0061] Furthermore, the centroid of a class can be calculated in several ways. One method involves using a sample of the elements of a class. From a mathematical point of view, this amounts to determining the centroid of a class by solving:

[0062] [Math.Eq(ll)]

[0063] Another way is to determine the difference between an estimate of the centroid at a first time and an estimate of the centroid at a second time. From a mathematical point of view, this amounts to solving:

[0064] [Math.Eq(12)]

[0065] Figure 4 illustrates experimental results obtained on the CIFAR10 dataset. First, we evaluate how the function of The discriminant loss function according to the invention improves class compactness and inter-class separability compared to softmax loss. This is achieved by first training the network on the CIFAR10 training set. Next, the test set features are calculated. Finally, the t-SNE technique (Van der Maaten and Hinton 2008) is used to visualize the features in a two-dimensional space. As seen in the upper left of the figure, softmax loss does not allow for proper separation of classes in feature space. Conversely, as seen in the upper right of the figure, the discriminant loss function according to the invention provides better inter-class separability.

[0066] Next, we evaluate how the discriminant loss function according to the invention helps the network to generalize better to the test set by examining the test accuracy. We use only a portion of the training set for training, then evaluate on the complete test set, with the softmax loss and the discriminant loss function according to the invention. The results are presented in the table reproduced in the lower part of the figure, with the softmax loss results (L s) in the second row of the table and the results of the discriminant loss function according to the invention in the third row of the table (Ldiscr). It can be seen that at different percentages of the training set, the method according to the invention consistently outperforms the softmax loss. Moreover, with only 60% of the training set, the method according to the invention already outperforms the softmax loss using the full training set. This result proves that the discriminant loss function according to the invention helps the network to generalize better using less training data.

[0067] Figure 5 illustrates the same experiments performed on the SVHN dataset. As seen in the figure, comparing the softmax loss results on the left and the discriminant loss function results on the right, the results show the same results. According to the invention, improved separation and compactness are achieved. Furthermore, as shown in the table illustrated in the lower part of the figure, the method according to the invention consistently outperforms the softmax loss method. Remarkably, with only 40% of the training data set, the method according to the invention already achieves the same test accuracy as the softmax network using the complete training data set. This result demonstrates that the discriminant loss function helps the network generalize more effectively using less training data.

[0068] As previously mentioned, the discriminant loss function according to the invention is applied to the penultimate layer of the network (just before the softmax layer). Neural networks are generally composed of layers that can be grouped into two subsets. The first subset of layers is dedicated to feature extraction, and the second subset, dedicated to classification, comprises the last layer(s) of the neural network. It is this second subset that is specifically referred to when discussing the softmax layer. Therefore, it is interesting to examine the impact of this loss function on the overall network, as the results presented in Figures 4 and 5 correspond precisely to the feature of the penultimate layer.Therefore, it can be assumed that the discriminant loss function according to the invention only impacts the fully connected layers that provide discriminating features and has little or no impact on the feature extractor. This is why the same t-SNE technique is applied to the intermediate layers of the feature extractor. In the two experiments above, ResNetl8 (He et al. 2016) was used as the feature extractor, followed by several fully connected layers before the softmax layer. ResNetl8 comprises four main blocks, each producing outputs of dimension H x W x Ci, where H and W are spatial dimensions (and thus depend on the input dimension), and Ci is the number of channels in block i (independent of the input dimension). The global means along the dimensions are first pooled. spatial analysis is performed to obtain a feature vector of dimension Ci (for each block i). Then, the t-SNE is performed as for the penultimate layer. The results for CIFAR10 and SVHN are shown in Figure 6, with the CIFAR10 dataset at the top and the SVHN dataset at the bottom.

[0069] For both datasets, we observe that there are no distinct clusters in the first two blocks. This is expected because these are shallow layers and no semantic features are actually captured. However, starting with block 3, the discriminant loss function according to the invention appears to help the network produce more distinctive clusters compared to softmax loss. This effect is also clearly observed for block 4 in both datasets. This implies that the discriminant loss function according to the invention not only impacts the features of the penultimate layer but also helps the network learn more discriminating features at intermediate levels. In conclusion, the discriminant loss function according to the invention is useful for network training.

[0070] Therefore, thanks to the method and device according to the invention described above, a solution is provided for training a neural network that can be applied to any deep learning neural network. A neural network thus trained can be used, in particular, for various classification tasks, for example, but not limited to, facial recognition, traffic sign identification, driver behavior recognition, road surface type identification, speech recognition, etc.

Claims

DEMANDS:

1. A method for managing, by a computer device (100), the training of a neural network, characterized in that said method comprises the steps of: i) determining a first loss value by means of a discriminant loss function applied to the penultimate layer of said network, said discriminant loss function comprising a compactness term, configured to guarantee the compactness of each class, a separability term, configured to guarantee the separability of each class, and a regularization term, configured to prevent the centroids of the classes from being pushed to infinity; ii) determining a second loss value by means of a softmax loss function applied to the last layer of said network; iii) determining a third loss value, resulting from the combination of the first loss value and the second loss value.

2. A method according to claim 1, characterized in that said compactness term is established as a function of the distance L2 between a sample of a class and the centroid of the class and as a function of a first predefined parameter, said compactness term being adapted so that said distance L2 is less than said first predefined parameter.

3. A method according to any one of the preceding claims, characterized in that said separability term is determined as a function of a maximization of the distance between a centroid of a class and a decision boundary and as a function of a second predefined parameter, said separability term being adapted to impose that each distance between a centroid of a class and the decision boundaries of the classes is greater than the second predefined parameter.

4. A method according to any one of the preceding claims, characterized in that said regularization term is obtained by determining the centroid of a class using a sample of points from said class, or by determining a variation between an estimate of the centroid of a class at a first instant and an estimate of the centroid of a class at a second instant.

5. Device (100) for managing the training of a neural network, characterized in that the device comprises at least one information processing unit (101), comprising at least one processor, and a data storage medium (102) configured to implement a method according to any one of the preceding claims.

6. Computer program comprising program code instructions for executing the steps of a process according to any one of claims 1 to 4 when said program is executed by at least one processor.

7. A medium usable in a computer, characterized in that a program according to claim 6 is stored thereon.

Images