Method of training an image classification model

Abstract

A computer-implemented method of training a neural network for classifying an image into one of a plurality of classes, the method comprising: extracting, from the neural network, a plurality of sub-class center vectors for each class; inputting an image into the neural network, wherein the image is associated with a predefined class; generating, using the neural network, an embedding vector corresponding to the input image; determining a similarity score between the embedding vector and each of the plurality of sub-class center vectors; updating parameters of the neural network using an objective function in accordance with a plurality of the similarity scores; extracting a plurality of updated parameters from the neural network; and updating each sub-class center vector in accordance with the extracted updated parameters.

Description

TECHNICAL FIELD

[0001] The present invention relates to training a neural network for image classification, and in particular to a method of training a neural network for image classification on a dataset with noisy labels. BACKGROUND

[0002] Image representations using deep convolutional neural network (DCNN) embeddings with margin penalty are the preferred method for existing image classification (e.g. face recognition). DCNNs typically map images to feature embeddings that achieve intra-class compactness and inter-class variability after a step of pose normalisation.

[0003] To train a DCNN, each image in a dataset must be labelled as related to an identity class. Each identity class is intended to contain only images of a particular type / category. There are several techniques available to label each image; however, such techniques can result in incorrect labelling, known as label noise. SUMMARY

[0004] The present specification discloses a neural network system for achieving image classification that is more robust to label noise.

[0005] According to a first aspect of the present invention, there is provided a computer- implemented method of training a neural network for classifying an image into one of a plurality of classes, the method comprising: extracting, from the neural network, a plurality of sub-class centre vectors for each class; inputting an image into the neural network, wherein the image is associated with a predefined class; generating, using the neural network, an embedding vector corresponding to the input image; determining a similarity score between the embedding vector and each of the plurality of sub-class centre vectors; updating parameters of the neural network in accordance with a plurality of the similarity scores using an objective function; extracting a plurality of updated parameters from the neural network; and updating each sub-class centre vector in accordance with the extracted updated parameters.

[0006] The method can further comprise, prior to updating the parameters of the neural network: inputting another image into the neural network, wherein the image is associated with a predefined class; generating, using the neural network, another embedding vector corresponding to the input image; determining another similarity score between the other embedding vector and each of the plurality of sub-class centre vectors; and wherein updating the parameters of the neural network is further based on the other similarity score.

[0007] The method can further include determining, using the similarity scores, a closest sub-class center vector for each class, wherein the objective function includes a multi-center loss term that compares similarity scores between the embedding vector and the closest sub-class center vector in the predefined class to similarity scores between the embedding vector and the closest sub-class center vector in every other class. The multi-center loss term can be a margin-based softmax loss function. The embedding vector and each sub-class center vector can be normalized, and the similarity score can be an angle between the embedding vector and the sub-class center vector.

[0008] Each class can include a dominant sub-class, and the method can further include, for each class, determining an intra-class similarity score between a dominant sub-class center vector and each other sub-class center vector in the class, wherein the objective function includes an intra-class compactness term that uses the intra-class similarity score.

[0009] Each sub-class center vector can be normalized, and the intra-class similarity score can be an angle between the dominant sub-class center vector and another sub-class center vector in the class.

[0010] The neural network can include a plurality of connected layers, and each sub-class center vector can be updated using updated parameters extracted from a last fully connected layer of the neural network.

[0011] Each class can include a dominant sub-class, and the method can further include discarding a non-dominant sub-class from a class if a distance between the sub-class center vector of the non-dominant sub-class and the dominant sub-class center vector exceeds a threshold distance.

[0012] Discarding a non-dominant sub-class from a class can be performed according to a threshold condition being met. The threshold condition can be exceeding a first threshold number of training rounds. The method can further include discarding all non-dominant sub-classes according to another threshold condition being met. The other threshold condition can be exceeding a second threshold number of training rounds.

[0013] The images can be face images.

[0014] The classes can correspond to classification conditions of the images, and the images can be from a batch that contains label noise, such that the batch can include at least one image that is labeled as relating to a class that does not correspond to a classification condition of the at least one image.

[0015] According to another aspect of the present application, there is provided a computer-implemented image classification method, the method comprising: inputting a first image to a neural network to map the first image to an embedding vector; identifying, using the neural network, a class corresponding to the first image based on the embedding vector, wherein the neural network has been trained using any of the methods disclosed herein.

[0016] According to another aspect of the present application, there is provided a system comprising: one or more processors; and a memory comprising computer-readable instructions that, when executed by the one or more processors, cause the system to perform any of the methods disclosed herein.

[0017] According to another aspect of the present application, there is provided a computer program product comprising computer-readable instructions that, when executed by a computing device, cause the computing device to perform any of the methods disclosed herein. BRIEF DESCRIPTION OF DRAWINGS

[0018] Certain embodiments of the present application will now be described, by way of example, with reference to the following drawings.

[0019] Figure 1 A schematic diagram showing a method of generating embedding vectors from samples;

[0020] Figure 2 A schematic diagram showing a method of training a neural network for image classification using a multi-centre based objective function;

[0021] Figure 3 A schematic diagram showing a multi-centre based objective function;

[0022] Figure 4 A comparison of triplet loss, tuple loss, ArcFace loss and multi-centre based loss;

[0023] Figure 5 A schematic diagram showing sub-classes of a particular class;

[0024] Figure 6 A flowchart showing a method of training a neural network for image classification;

[0025] Figure 7 A flowchart showing a method of training a neural network for image classification;

[0026] Figure 8 A flowchart showing a method of training a neural network for image classification;

[0027] Figure 9 An example of a computer system for performing face recognition is shown. DETAILED DESCRIPTION

[0028] In the embodiments described herein, the image classification neural network (e.g. a face recognition neural network) is trained using sub-class center vectors derived from the neural network in addition to the embedding vectors output by the neural network.

[0029] Using the sub-class center vectors to form a multi-center based loss function can enable a neural network trained on noisy data to have higher image classification performance compared to other methods of training a neural network on noisy data. For example, the image classification neural network can have a lower error rate. It can also allow training of the neural network to be performed directly on the original dataset, thereby reducing the need for resource intensive manual cleaning of the original dataset.

[0030] In the following, embodiments will be described in connection with face images (e.g. the image classification task is face recognition). However, it should be appreciated that the methods can also be applied to other image classification tasks, such as medical image classification, handwritten digit classification, and object image classification. Figure 1 A schematic diagram showing a method 100 of generating an embedding vector 106 from an input image 102 is shown. The input image 102 can comprise a face image. The input image 102 is input to a neural network 104 which processes the input image 102 through a plurality of neural network layers to generate an embedding vector 106 which encodes features and / or attributes in the input image 102. The embedding vector 106 is used to classify the image into one of a plurality of image classes 108 according to the values of the components of the embedding vector 106.

[0031] The input image 102 (I) comprises a set of pixel values in an array (e.g. a two-dimensional or three-dimensional array). For example, in a color image In some embodiments, the image I can be a black and white / gray scale image. In some embodiments, the image I can be a color image.

[0032] The neural network 104 includes multiple layers of nodes, each node associated with one or more parameters. The parameters of each node of the neural network 104 can include one or more weights and / or biases. A node takes as input one or more outputs of nodes in a previous layer. The one or more outputs of nodes in the previous layer are used by the node to generate an activation value using an activation function and the parameters of the neural network. One or more layers of the neural network 104 can be convolutional layers. One or more layers of the neural network 104 can be down-sampling (pooling) layers. One or more layers of the neural network 104 can be fully connected (FC) layers. One or more layers of the neural network 104 can perform batch normalisation (BN). One or more layers of the neural network 104 can be dropout layers.

[0033] In some embodiments, the neural network 104 can include a residual network. For example, the neural network can include ResNet-100 or DenseNet with a BNDropout-FC-BN structure after the last convolutional layer. For example, the last layer can generate a 256-D, 512-D or 1024-D embedding vector.

[0034] For an input image 102(i), an embedding vector 106 is an M-dimensional vector, i.e., it includes multiple components. The embedding vector is an encoded representation of the input image 102. The embedding vector can be normalised. The embedding vector 106 can be used to classify the image into a class 108(j). The class 108 is one of multiple possible classes 108(C). In the embedding space, each class is associated with a class centre vector 110 The embedding vector 106 is compared to the class centre vector 110 of each available class 108 in order to classify the input image 102 into one of the available classes 108. For example, the input image 102 is classified into a class 108 by the class centre vector 110 that is closest to the embedding vector 106 derived by the neural network 104 from the input image 102.

[0035] Each of the multiple classes 108 can be related to a particular feature of the input image 102. For example, the class 108 can be related to an identity of an input human face image 102. The class 108 can be related to whether the input image 102 contains an indication of a medical condition. The class 108 can be related to a value of a handwritten digit in the input image 102. The class 108 can be related to a type of object contained in the input image 102.

[0036] Each class 108(j) can contain multiple (K) sub-classes. In the embedding space, each sub-class is associated with a sub-class centre vector (w jkThe class center vector 110 of the class 108 can be based on one or more sub-class center vectors of the class 108.

[0037] For example, the class center vector 110 can be a sub-class center vector of a dominant sub-class in the class 108.

[0038] With the training methods discussed below, each sub-class can be related to a specific feature of the input image 102. The specific feature can include a specific feature of the class 108 and include at least one additional feature. For example, a sub-class can be related to the identity of an input face image 102 and whether the person wears glasses in the input face image 102. A sub-class can be related to an input image 102 containing an indication of a medical condition and the type of medical condition. A sub-class can be related to the value of a handwritten digit in an input image 102 and features about the shape of the handwritten digit. A sub-class can be related to the type of object contained in an input image 102 and the color of the object.

[0039] Since the locations of the sub-class center vectors in the embedding space are discovered by the training methods discussed below, the specific features of the input image 102 related to the sub-classes can only become apparent after the training methods are completed. The specific features can not be precisely determined, but can be classified as follows: the input image 102 is correctly labeled as belonging to the class 108 or the input image 102 is incorrectly labeled as belonging to the class 108. The specific feature of the input image 102 can be that the input image 102 is correctly labeled as being related to a particular feature of the class 108, and each class can have one or more than one sub-class related to the specific feature. The specific feature of the input image 102 can be that the input image 102 is incorrectly labeled as being related to a particular feature of the class 108, and each class can have one or more than one sub-class related to the specific feature.

[0040] Figure 2 A schematic diagram showing a method 200 of training a neural network for image classification using a multi-center based objective function is shown. Images 202 from a training batch 208 are input to a neural network 204. The neural network 204 processes the input images through a series of layers and outputs embedding vectors 206. In addition, sub-class center vectors 210 are also extracted from the weights in one or more final layers of the neural network 204. An objective function 212 is evaluated based on the embedding vectors 206 and the extracted sub-class center vectors 210 and used to determine parameter updates 214 for the neural network 204.

[0041] The training batch 208 includes at least one image 202. The batch 208 can include multiple images. Each image in the batch 208 is labeled as relating to one of a plurality of predefined classes. The batch 208 can be the entire training dataset, or a subset of the training dataset. The batch 208 and / or the training dataset can contain label noise, e.g., include one or more mislabeled images.

[0042] A plurality of sub-classes is defined for each class according to weights extracted from the neural network 204. The weights define a sub-class center vector 210 for each sub-class.

[0043] A target function 212 (also referred to herein as a loss function) is used to train the parameters of the neural network 204. The loss function can be viewed as a measure of error of the neural network 204, and the goal of the training 200 is to minimize / approximately minimize the loss function on the training dataset. The target function 212 can include a function based at least in part on a comparison involving the sub-class center vectors 210 of the sub-classes.

[0044] The initial number K of sub-classes for each class can be set by setting the parameters of the neural network 204. The initial number K of sub-classes for each class is set by setting the dimensionality of the last layer (or layers) of the neural network 204. The initial number K of sub-classes for each class should be large enough to separate noisy data from clean data. The initial number K of sub-classes for each class can be selected based on experimental results to determine the value of K that trains the neural network 204 to be most accurate. The initial value of K can be between 3 and 15, such as between 3 and 10, e.g., 10, 5, or 3. On a typical noisy training dataset such as Celeb500K, setting the value of K to 3 can improve the performance of the trained neural network 204 compared to setting the value of K to 5 or 10.

[0045] The parameters of the neural network 204 can be updated after each image 202 has been input to the neural network 204. Alternatively, the parameters of the neural network 204 can be updated after a plurality of images 202 have been input to the neural network 204 and the value of the target function for each image 202 has been determined. For example, the parameters of the neural network 204 can be updated after the batch 208 has been input to the neural network 204. The target function can be averaged over the batch 208.

[0046] An update 214 to the neural network 204 is determined according to the target function 212. An optimization procedure, e.g., stochastic gradient descent, can be applied to the target function 212 in order to determine the update 214.

[0047] An epoch refers to one cycle through the entire training dataset. For example, an epoch can correspond to one forward pass and one backward pass through the training dataset. An epoch can correspond to each image 202 in the training dataset being input to the neural network 204 at least once. The learning rate of the update 214 can vary throughout the training process. For example, the learning rate can start at a first value (e.g., 0.1) and be divided by one or more predetermined factors at a threshold number of iterations, e.g., by ten at ten and sixteen epochs. The training 200 can stop after a threshold number of epochs is completed. The threshold number of epochs can be in the range [20, 40], preferably in the range [20, 30], e.g., at 30, 20, or 30 epochs. In embodiments where the optimization process uses a momentum-based method, the momentum of the update 214 can be set to a value in the range [0.5, 0.99], preferably in the range [0.85, 0.95], e.g., 0.9. Furthermore, a weight decay can be set to a positive number less than 1, preferably much less than 1, e.g., 5e -4 .

[0048] Each image 202 in the batch 208 is labeled as being associated with one class, and the feature embedding vector 206 derived from the image 202 is labeled as being associated with the same class. In the embedding space, each class contains feature embedding vectors 206 that are labeled as being associated with the class.

[0049] A sub-class in a class can include noise data, specifically, a sub-class can include only noise data. For example, a sub-class can contain feature embedding vectors 206 that are mislabeled. A feature embedding vector 206 in a class is located in a sub-class of the class that has a sub-class center vector 210 closest to the feature embedding vector 206. To remove noise data from the training dataset, the neural network 204 can discard sub-classes that are more than a threshold distance away from a dominant sub-class. The threshold distance can be based on a difference between a sub-class center vector 210 of a sub-class and a sub-class center vector 210 of a dominant sub-class. The threshold distance can be an angular difference in the range [60°, 90°], preferably in the range [70°, 80°], e.g., 70°, 75°, or 80°. The dominant sub-class of a class is the sub-class in the class that contains the most feature embedding vectors 206.

[0050] After the neural network 204 has discriminative power, the neural network 204 can discard non-dominant sub-classes that satisfy a threshold condition and continue training on the remaining sub-classes. It can be assumed that the neural network 204 has sufficient discriminative power after a threshold number of epochs is completed, e.g., in the range [10, 20], such as 15 epochs are completed.

[0051] The update parameter can produce a new set of weights in the last layer (or layers) of the neural network 204, thereby updating the sub-class center vectors 210.

[0052] Figure 3 An illustration of a method 300 of determining a multi-center based objective function 310 is shown. The similarity scores 306 between the embedding vector 302 output by the neural network and a set of sub-class center vectors 304 are determined. Each similarity score 306 is converted to an angular representation 308. The multi-center based objective function 310 is computed from the angular representation 308 of the similarity scores 306. The parameters of the neural network are updated using the objective function 310. The objective function 310 can have the effect of compressing the sub-classes 314 and / or compressing the sub-centers 316 of the classes.

[0053] The sub-class center vectors 304 can be extracted from the last layer (or multiple layers) of the neural network. For example, the sub-class center vectors 304 can be extracted from the last fully connected (FC) layer of the neural network. The last layer of the neural network can be a linear layer, which can also be referred to as a weight matrix. The sub-class center vectors 304 can be represented as a tensor where N is the number of classes, K is the number of sub-classes for each class, and M is the dimension of the feature embedding vector 302, i.e., each of the K sub-class center vectors 304 for each class is a vector of dimension M. Each of the sub-class center vectors 304 can be normalized.

[0054] Before training, the parameters of the neural network can be randomly initialized. As a result, the sub-class center vectors 304 can also be set with random initialization.

[0055] The similarity score 306 between the embedding vector 302 (x i ) and the sub-class center vector 304 (w jk ) is a measure of the difference between the embedding vector 302 and the sub-class center vector 304. The similarity score 306 between the current embedding vector 302 (x i ) and all NK sub-class center vectors 304 can be represented as a matrix The current embedding vector 302 (x i ) corresponds to the feature embedding vector 302 (x i ) of the latest input image. The similarity score 306 between the embedding vector 302 and the sub-class center vector 304 can be computed in a variety of ways. One example is to take the dot product between the embedding vector 302 and the sub-class center vector 304, i.e.,

[0056]

[0057] For example, the similarity score 306 can be computed as Norms are used to normalize the embedding vector 302 and the subclass center vector 304. Another example is to use a pooling step. Other examples include Euclidean distance between vectors, absolute value norm, Euclidean norm, Taxicab norm, and / or p-norm.

[0058] In some embodiments, the similarity score 306 can be converted into an angle representation 308 for use in the multi-center-based objective function 310. The angle representation 308 provides the angle between the current embedding vector 302 and the center vector 304 of each subclass. For example, the angle representation 308 can be determined by taking the arccosine of the similarity score 306.

[0059]

[0060] For each class j, the subclass center vector 306 that is “closest” to the embedding vector 302 is determined based on the similarity score 306 (or, in some embodiments, based on its angular representation 308). For example, the closest subclass center vector 304 can be determined by finding the minimum value of θi,jk for that class, i.e.:

[0061]

[0062] The loss function 310 can be evaluated using the set of the nearest subclass center vectors 304 for each class, from which parameter updates can be determined.

[0063] Objective function 310 may include a multicenter loss term L The multicenter-based loss function 310 may include a comparison of similarity scores 306 between the embedding vector 302 associated with a specific class and the nearest subclass center vectors 304 of all classes. In other words, the multicenter loss term may compare the distance from the embedding vector 302 to the nearest subclass center vector 304 of a predefined class with the distance to the nearest subclass center vector 304 of each other class.

[0064] The multicenter loss term can be an additive angular margin loss. For example, the multicenter loss term could be a margin-based softmax function applied relative to the nearest subclass center vector 304 for each class. An example of such a multicenter loss term based on the ArcFace loss function is derived as follows:

[0065]

[0066] Where m is the margin penalty, s is the feature scale, and y is the feature size. iis a predefined class label for the input image from the training dataset. The margin loss can be in the range [0.1, 1], preferably in the range [0.3, 0.7]. For example, the margin loss can be 0.5. The feature scale can be in the range [16, 128], preferably in the range [32, 128]. For example, the feature scale can be 64.

[0067] While the above term L based on the multi-center based softmax loss function 310 is based on ArcFace, it should be understood that this is merely an example, and the present invention is not limited to using ArcFace. The method of training a neural network using a multi-center based softmax loss can be applied to any method that uses a softmax loss that compares a sample to a class center vector. For example, the method of training a neural network using a multi-center based softmax loss function can be applied to SphereFace loss and CosFace loss in a similar manner as described above for ArcFace loss.

[0068] The multi-center based loss function 310 causes the sub-class centers (sub-class center vectors) 304 to be distributed compactly, which is beneficial for inter-class comparison because more negative sub-class centers can approximate the distribution of all negative samples. Given an initial distribution 312 of K sub-class centers for each class in N classes, the multi-center loss term compresses the sub-classes 314 such that the samples in each sub-class are more tightly clustered around their sub-class center vector 304. The multi-center loss term can be averaged over the samples from the training batch before determining the update.

[0069] The method of training a neural network using a multi-center based softmax loss as described above is able to automatically discover the sub-class centers (sub-class center vectors) 304 in each class. Compared to ArcFace, the method of training a neural network using a multi-center based softmax loss function improves the robustness of two aspects of the margin loss: 1) pushing the anchor towards the positive center, 2) pushing the anchor away from the negative center.

[0070] In some embodiments, the objective function 310 can also include an intra-class compactness term L The intra-class compactness term compares the sub-class center vectors 304 in each class to the dominant sub-class center 304 of their class. The intra-class compactness term penalizes the deviation of the sub-class center vectors 304 in each class from the dominant sub-class center 304 of their class, thereby causing the sub-class center vectors 304 to cluster around the dominant sub-class of their class (i.e., sub-center compression 316). One example of such an intra-class compactness term can be derived by:

[0071]

[0072] where k' is the dominant sub-class of class j, θ jk′ ,jk is the angle between the sub-class center vector w jk and the dominant sub-class center vector w jk' .

[0073] The dominant sub-class of each class is the sub-class that contains the majority of images (or embedding vectors 302 corresponding to images) in the class. It can be assumed that the dominant sub-class contains accurately labeled samples. In other words, it can be assumed that the dominant sub-class of each class contains clean data. An image associated with a class is located in the sub-class of the class that has the sub-class center vector 304 closest to the embedding vector 302 corresponding to the image.

[0074] The objective function 310 can include a combination of the multi-center loss term and the intra-class tightness term. The multi-center loss term can be averaged over samples from a training batch before determining an update. One example of such a combined loss function is derived by:

[0075]

[0076] where B is the batch size and λ is a hyperparameter that controls the relative importance of the intra-class tightness term relative to the multi-center loss term. For example, the batch size can be 512 samples.

[0077] Figure 4 A comparison 400 of embodiments of triplet loss, meta-loss, “ArcFace” loss, and multi-center based loss is shown. There are two main research directions for training deep convolutional neural networks (DCNNs) for image classification (e.g., face recognition): training a multi-class classifier that can separate different identities in a training dataset, such as using a softmax classifier, and directly learning an embedding, such as triplet loss. Both softmax loss based methods and triplet loss based methods can achieve good image classification performance on large-scale training datasets and sophisticated DCNN architectures.

[0078] Triplet loss 402 simultaneously enforces intra-class tightness and inter-class difference for a triplet (x i , x j , x k ). Triplet loss enforces a clear Euclidean distance margin m between samples in the same class than between samples in different classes. Specifically, triplet loss compares the difference between a sample (x i ) 404 in a certain class and a sample (x j ) 406 in the same class to the difference between a sample (x i ) 404 in another class and a sample (xk The differences between 408 samples are compared. The differences between samples can be based on the deep feature embedding of the samples. An example of triplet loss is shown below:

[0079]

[0080] Where, x i and x j From the same category, x k From another class, m is the Euclidean distance margin.

[0081] Since the total number of triples can be the cube of the number of samples, the triple loss 402 method converges slowly and often requires costly data sampling techniques to provide useful triples to speed up training.

[0082] Tuple loss 410 collects multiple negative samples to avoid costly hard negative data mining as in triple loss 402. Tuple loss 410 compares the differences between anchor (sample) 404 and positive sample 406 with the anchor with the differences between anchor 404 and multiple negative samples 412. For example, for (N+1)-tuple training samples {x} i ,x1,···,x yi ,···,x N}, x yi It is x i Positive samples, {x j} N j = 1, j≠yi These are negative samples. The (N+1)-tuple loss is defined as follows:

[0083]

[0084] When {x j}N j When N = 1 is considered as the weight vector, the tuple loss 410 is similar to the multi-class logistic loss (i.e., softmax loss) formula. When N = 2, the corresponding (2+1)-tuple loss is very similar to the triple loss because there is only one negative sample for each pair of inputs and positive samples. When N > 2, the (N+1)-tuple loss is a partition function estimate of the ideal (L+1)-tuple loss, where the (L+1)-tuple loss is coupled with all negative samples. The (N+1)-tuple loss approximates the triple loss more accurately because the larger the value of N, the more accurate the approximation.

[0085] The margin-based Softmax 414 compares the difference between the anchor (sample) 404 and its corresponding weight vector (i.e., the current class center vector of the sample's class) 406 with the difference between the anchor and the non-corresponding weight vector (i.e., the current class center vector of the class not to which the sample belongs) 416. The margin-based Softmax penalizes this difference in the softmax loss. The margin-based Softmax can be defined as follows:

[0086]

[0087] in, Indicates weight The j-th column, where N is the number of classes.

[0088] For W j and x i application Normalization or any other suitable type of normalization, Where θ j It is the weight W j and embedding vector x i The angle between them is then the CosFace loss is:

[0089]

[0090] Where s is the scaling factor. Compared to (N+1)-tuple loss, the margin-based softmax method extensively uses all negative classes for comparison instead of a subset of negative samples, and effectively provides a proxy representation (W) for each class from online feature embeddings from GPU memory rather than all samples. j Create an index.

[0091] ArcFace also features an additive angular margin loss. ArcFace incorporates the margin *m* into the similarity metric between vectors. Specifically, ArcFace includes an angular margin *m*, which has a more explicit geometric meaning. For example, the ArcFace loss can be defined as follows:

[0092]

[0093] It not only has better performance, but also has a more explicit geometric meaning.

[0094] Since the triplet loss, the meta loss, the CosFace loss, and the ArcFace loss methods assume the existence of a clean dataset, all of these methods need to be trained on a carefully pre-processed clean dataset, which requires a lot of human effort. Images with label noise greatly reduce the recognition accuracy of these methods. Label noise occurs when a sample 404 is labeled as related to a class to which it does not belong. In other words, each sample 404 has a representation feature and each class label corresponds to a representation feature. A noisy label is a label corresponding to a representation feature and attached to a sample with a different representation feature. Large-scale datasets, especially face image datasets, often contain noise that can be modeled as label noise. Typically, there are two types of label noise: one is open set label noise, i.e., the true label is not in the training label dataset but is incorrectly labeled as a sample in the dataset; the other is closed set label noise, i.e., the true label is in the training label dataset but is incorrectly labeled.

[0095] To implement a more robust image classification neural network that is less susceptible to label noise, embodiments of the present invention relate to a method of training a neural network using a multi-center based softmax loss function. The method includes setting multiple sub-classes for each class. In particular, embodiments of the present invention relate to a method of training a neural network for image classification using a multi-center based ArcFace loss function, however it should be understood that the present invention is not limited to application to the ArcFace loss function. In particular, the method can be applied using any softmax loss function.

[0096] According to one embodiment, the multi-center based softmax loss 418 includes a comparison between a similarity score between the embedding vector 404 associated with a particular class and a sub-class center vector of all classes. The multi-center based softmax loss function 418 can include a comparison between a similarity score between the embedding vector 404 associated with a particular class and a sub-class center vector of all classes, where each class is represented by one of the plurality of sub-class center vectors associated with the class. For example, the multi-center based softmax loss function 418 can include a comparison between a similarity score between the embedding vector 404 associated with a particular class and a sub-class center vector 420 of the class that is closest to the embedding vector 404 and a similarity score between the embedding vector 404 and a sub-class center vector 422 of another class. Further, the multi-center based softmax loss function 418 can include a comparison between a similarity score between the embedding vector 404 associated with a particular class and a sub-class center vector 420 of the class that is closest to the embedding vector 404 and a similarity score between the embedding vector 404 and a sub-class center vector 422 of another class that is farthest from the embedding vector 404. Further, the multi-center based softmax loss function 418 can include a comparison between a similarity score between the embedding vector 404 associated with a particular class and a sub-class center vector 420 of the class that is closest to the embedding vector 404 and a similarity score between the embedding vector 404 and a sub-class center vector 422 of another class that is closest to the embedding vector 404.

[0097] Figure 5 An illustration of sub-classes in the embedding space of a particular class 500 is shown. A single class 500 in a dataset is visualized after training using a multi-center based ArcFace loss. Training a neural network using a multi-center based ArcFace loss function causes the neural network to cluster images, which in turn causes hard samples 502 and noisy samples 504 to separate from the dominant clear and clean samples 506. If the number of sub-classes is too large for a class, some sub-classes can be empty 508.

[0098] One advantage of using a multi-center based loss is that noisy samples can be discarded into non-dominant sub-classes, thus the method is more robust to training noisy data.

[0099] Similar to ArcFace, the multi-center based softmax loss function has two objectives: 1) to push the anchor (i.e., embedding vector of an image in a class) towards the positive center (i.e., closest sub-class center of the class); 2) to push the anchor (i.e., embedding vector) away from the negative center (i.e., sub-class center of other classes that is closest to the embedding vector of the image).

[0100] We can assume that clean data dominates in each class. If clean data dominates in each class, training a neural network using a multi-center-based softmax loss function will likely result in the data being partitioned into a dominant subclass containing both clean and sharp images and multiple non-dominant subclasses containing hard or noisy images, for example, as... Figure 5 As shown. This occurs because the first aspect of the method of training a neural network using a multi-center-based softmax loss function aims to push the anchor point closer to one of several positive subclass centers, but not all. Specifically, the anchor point is pushed closer to the subclass center within the same class closest to the anchor point. If the anchor point is a noisy image 504 or a hard image 502, it is difficult to push the anchor point to the dominant subclass center. Note that it is relatively easy to push most clean samples to the dominant subclass. Therefore, noisy images may only affect and / or form non-dominant subclasses, not dominant subclasses. Thus, the method of training a neural network using a multi-center-based softmax loss function is robust to open-set label noise.

[0101] For closed-set label noise, its impact on the first objective of methods training neural networks using a multi-center-based softmax loss function is similar to that of open-set label noise discussed above. For the second objective, closed-set label noise can be... Modeling is performed, in which, Let Y and Y represent the observable noise labels and potential true labels, respectively, random variables. Probability This represents the noise level; the higher the probability, the lower the reliability of the noise data. Let... Represents a random variable that supplements the subclass label, for example, Specifies the probability that image x does not belong to subclass i.

[0102] The second objective of the method 200, which uses a multi-center-based loss function to train a neural network, is to transform noisy data into supplementary data with K(N-1) supplementary subclasses. Specifically, given a noise anchor point (x, y... ～ Instead of directly fitting... The second objective of the method 200, which uses a multi-center-based loss function to train a neural network, is to generate and fit supplementary data. i = 1, ..., K(N-1), where i is the index of all subclasses of the non-positive class. The supplementary data consists of subclass labels. Since the supplementary data is generated equally for each subclass, we can obtain... Similarly, it can be used The higher the probability, the lower the reliability of the supplementary data.

[0103] As the number of centers / sub-class centers increases, the augmented data will be more reliable than the noisy data, as shown below.

[0104] The augmented data can be obtained

[0105]

[0106] where the second equality holds because the second aspect objective based on the multi-center softmax loss will push the noisy data points to the K(N-1) augmented labeled data points. Furthermore, it can also be obtained that

[0107]

[0108] where Y denotes the true class label, denotes the noisy class label, denotes the sub-class label of the noisy anchor point (x,y ～ ):

[0109]

[0110] More specifically,

[0111]

[0112] Equation 1 indicates that using the augmented data will be more reliable than using the noisy data directly. Equation 2 implies that as the number of centers / sub-classes increases, i.e., K increases, the gap between the noise level of the augmented data and the noisy data will become larger, which means that as the number of centers / sub-classes increases, the augmented data will be more reliable than the noisy data.

[0113] Figure 6 to Figure 8 Flowcharts illustrating various methods of training a neural network for image classification using a multi-center based softmax loss function are shown. The methods can be implemented on a computer. The methods can correspond to using any of the multi-center based softmax loss functions described above. The methods can be performed on each image in one batch. The methods can be performed on each image in multiple batches, where each batch includes images from a training dataset.

[0114] In operation 6.1, a plurality of sub-class center vectors for each class are extracted from the neural network. The sub-class center vectors can correspond to the weights in the last layer (or multiple layers) of the neural network. The initial weights of the neural network can be randomly initialized.

[0115] In operation 6.2, a sample image labeled as related to a particular class is input to the neural network. The sample image is obtained from a training dataset that includes a plurality of images, each image labeled as belonging to a corresponding class.

[0116] In operation 6.3, a feature embedding vector is generated from the sample image using a neural network. The neural network processes the sample image through a plurality of layers and outputs an embedding vector representing the sample image. The embedding vector is labeled as belonging to the same class as the sample image.

[0117] In operation 6.4, a similarity score between the embedding vector and each of a plurality of sub-class center vectors is determined. That is, for each sub-class in each class, a similarity score between the embedding vector and a sub-class center vector is determined. The similarity score can be determined by taking a norm between the embedding vector and the sub-class center vector. The similarity score can be determined using a pooling layer. The similarity score can represent an angle between the embedding vector and the sub-class center vector. In some embodiments, the closest sub-class center vector for each class is determined using the similarity scores.

[0118] Operations 6.2 through 6.4 can be performed iteratively on a training batch to determine similarity scores between embedding vectors for each image in the batch and sub-class center vectors. In other words, each embedding vector generated by the neural network from the batch has a similarity score computed with each sub-class center vector.

[0119] In operation 6.5, parameters of the neural network are updated using the objective function based on the plurality of similarity scores between the embedding vector and the sub-class center vectors. The parameter updates can be determined by applying an optimization procedure, such as stochastic gradient descent, to the objective function.

[0120] The objective function can include a multi-center based loss term that compares the similarity score between the embedding vector labeled as related to a class and the closest sub-class center vector in that class to the similarity scores between the embedding vector and the closest sub-class center vectors in each other class. The multi-center based loss term can be a margin-based softmax loss function.

[0121] In some embodiments, the updates can be determined based on similarity scores for a batch of training data, rather than just a single training example.

[0122] In some embodiments, each class includes a dominant sub-class. The dominant sub-class is the sub-class in the class for which the most training examples are related, e.g., has the most sub-class center vectors closest to the training image embedding vectors. The dominant sub-class in each class can be updated during training. The method can further include, for each class, determining an intra-class similarity score between the dominant sub-class center vector and each other sub-class center vector in the class, wherein the objective function includes an intra-class tightness term using the intra-class similarity scores. The intra-class similarity score can be an angle between the dominant sub-class center vector and the other sub-class center vector.

[0123] In operation 6.6, updated parameters are extracted from the neural network to determine updated sub-class center vectors. The updated parameters can be extracted from the last layer (or multiple layers) of the neural network to define the updated sub-class center vectors.

[0124] In operation 6.7, a first threshold condition is checked. If the first threshold condition is satisfied, the training can be terminated, or the next training phase (as described below in connection with Figure 7 and Figure 8 ) can be entered. For example, if the first threshold condition is satisfied, the neural network can discard non-dominant sub-classes before terminating the training. If the first threshold condition is not satisfied, operations 6.2 to 6.7 are repeated using the updated sub-class center vectors. The first threshold condition can be completion of a predetermined number of rounds. For example, the first threshold condition can be completion of a number of rounds in the range [20, 50], preferably in the range [30, 40]. In particular, the first threshold condition can be completion of 30 rounds.

[0125] With reference to Figure 7 , operations 7.1 to 7.6 are the same as operations 6.1 to 6.6.

[0126] In operation 7.7, a threshold condition related to the discriminative power of the network is checked (referred to herein as a second threshold condition, to distinguish it from the first threshold condition described in connection with Figure 6 ). If the second threshold condition is satisfied, it indicates that the neural network has sufficient discriminative power. In other words, if the second threshold condition is satisfied, it can be determined that the neural network has been sufficiently trained to separate noisy data from clean data. If the second threshold condition is not satisfied, operations 7.2 to 7.7 are repeated using the updated sub-class center vectors. The second threshold condition can be completion of a particular number of rounds. For example, the second threshold condition can be completion of a predetermined number of rounds. The predetermined number of rounds can be in the range [10, 20], for example completion of 15 rounds.

[0127] In operation 7.8, the neural network discards sub-classes in a class whose sub-class center vector is further than a threshold distance from the dominant sub-class center vector of the class. The sub-classes can be discarded to remove them from the training dataset. Such a step actively removes data determined to be noisy, to prevent the noisy data from having a detrimental effect on the training of the neural network.

[0128] In operation 7.9, a termination threshold condition (also referred to herein as a third threshold condition) is checked. If the third threshold condition is satisfied, training can be terminated. If the third threshold condition is not satisfied, operations 7.2 to 7.9 are repeated using the updated sub-class center vectors. The third threshold condition can be completing a particular number of rounds. For example, the third threshold condition can be completing a number of rounds in the range [20, 50] (preferably, in the range [30, 40]). In particular, the third threshold condition can be completing 30 rounds. The third threshold condition can be the same as the first threshold condition.

[0129] Reference is made to Figure 8 Operations 8.1 to 8.8 can be the same as operations 7.1 to 7.8.

[0130] In operation 8.9, if the third threshold condition is satisfied, it is determined that the neural network has been sufficiently trained and enters the next training phase. If the third threshold condition is not satisfied, operations 8.2 to 8.9 are repeated using the updated sub-class center vectors. The third threshold condition can be completing a particular number of rounds. For example, the third threshold condition can be completing a number of rounds in the range [20, 50] (preferably, in the range [30, 40]). In particular, the third threshold condition can be completing 30 rounds. The third threshold condition can be the same as the first threshold condition.

[0131] In operation 8.10, the neural network discards the non-dominant sub-classes in the classes. Thus, each class contains only dominant sub-classes, and thus each class is represented by a dominant sub-class center vector.

[0132] In operation 8.11, training is terminated.

[0133] Figure 9 A schematic example of a system / apparatus for performing any of the methods described herein is shown. The system / apparatus shown is an example of a computing device. Those skilled in the art will appreciate that other types of computing devices / systems can alternatively be used to implement the methods described herein, for example a distributed computing system.

[0134] The apparatus (or system) 900 comprises one or more processors 902. The one or more processors control the operation of the other components of the system / apparatus 900. For example, the one or more processors 902 can comprise a general purpose processor. The one or more processors 902 can be a single core device or a multiple core device. The one or more processors 902 can comprise a central processing unit (CPU) or a graphical processing unit (GPU). For example, 8 NVIDIA Tesla P40 (24GB) GPUs.

[0135] Alternatively, the one or more processors 902 can include special purpose processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors can be included.

[0136] The system / device includes working memory or volatile memory 904. The one or more processors can access the volatile memory 904 to process data and can control storage of data in memory. The volatile memory 904 can include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or can include flash memory (e.g., an SD card).

[0137] The system / device includes non-volatile memory 906. The non-volatile memory 906 stores a set of operational instructions 908 in the form of computer readable instructions for controlling the operation of the processor 902. The non-volatile memory 906 can be any type of memory, such as read only memory (ROM), flash memory, or magnetic drive memory.

[0138] The one or more processors 902 are used to execute the operational instructions 908 to cause the system / device to perform any of the methods described herein. The operational instructions 908 can include code related to the hardware components of the system / device 900 (i.e., drivers), as well as code related to the basic operation of the system / device 900. Generally, the one or more processors 902 use the volatile memory 904 to execute one or more instructions of the operational instructions 908 (which are permanently or semi-permanently stored in the non-volatile memory 906) to temporarily store data generated during execution of the operational instructions 908.

[0139] The methods described herein can be implemented as digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These can include a computer program product (e.g., software stored on a disk, optical disk, memory, programmable logic device, etc.) including computer readable instructions that, when executed by a computer, for example, in conjunction with Figure 8 The described cause the computer to perform one or more of the methods described herein.

[0140] Any system features described herein can also be provided as method features, and vice versa. Alternatively, the device and functional features used herein can be expressed in terms of their corresponding structural counterparts. In particular, method aspects can apply to system aspects, and vice versa.

[0141] While certain examples describe samples as facial images, the present application is not limited to this embodiment. The above-described methods can be used to train a neural network to classify any type of data. Further, while certain examples describe classes as corresponding to the identity of an individual, the present application is not limited to this embodiment. Classes can correspond to any classification condition.

[0142] Furthermore, any, some and / or all features of one aspect can be applied to any, some and / or all features of any other aspect, by any suitable combination. It will be appreciated that particular combinations of the various features described and defined in any aspect of the application can be specifically designed for a particular application.

[0143] Although several embodiments have been shown and described, it will be understood by those skilled in the art that changes can be made in these embodiments without departing from the principles of the application, the scope of which is defined in the claims.

Claims

1. A computer-implemented method of training a neural network, the method comprising: For classifying an image into one of a plurality of classes, the method comprises: extracting a plurality of sub-class center vectors for each class from one or more last layers of the neural network; inputting an image into the neural network, wherein the image is associated with a predefined class; generating, using the neural network, an embedding vector corresponding to the input image; determining a similarity score between the embedding vector and each of the plurality of sub-class center vectors; determining, using the similarity scores, a closest sub-class center vector for each class; updating parameters of the neural network according to a plurality of the similarity scores using an objective function, the objective function comprising a multi-center loss term comparing a similarity score between the embedding vector and the closest sub-class center vector in the predefined class to a similarity score between the embedding vector and the closest sub-class center vector in each other class; extracting a plurality of updated parameters from the neural network; updating each sub-class center vector according to the extracted updated parameters.

2. The method of claim 1, wherein, The method further comprises, prior to updating the parameters of the neural network: inputting another image into the neural network, wherein the image is associated with a predefined class; generating, using the neural network, another embedding vector corresponding to the input image; determining another similarity score between the other embedding vector and each of the plurality of sub-class center vectors; wherein updating the parameters of the neural network is further based on the other similarity scores.

3. The method of claim 1, wherein, The multi-center loss term is a margin-based softmax loss function.

4. The method according to claim 1 or 2, characterized in that, The embedding vector and each sub-class center vector are normalized and the similarity score is an angle between the embedding vector and the sub-class center vector.

5. The method according to claim 1 or 2, characterized in that, Each class comprises a dominant sub-class, and the method further comprises: for each class, determining an intra-class similarity score between a dominant sub-class center vector and each other sub-class center vector in the class, wherein the objective function comprises an intra-class tightness term using the intra-class similarity scores.

6. The method of claim 5, wherein, Each sub-class center vector is normalized and the intra-class similarity score is an angle between the dominant sub-class center vector and another sub-class center vector in the class.

7. The method according to claim 1 or 2, characterized in that, The neural network comprises a plurality of connected layers, and each sub-class center vector is updated using updated parameters extracted from a last fully connected layer of the neural network.

8. The method of claim 1 or 2, wherein, Each class comprises a dominant sub-class, and the method further comprises discarding a non-dominant sub-class from the class if a distance between the sub-class center vector of the non-dominant sub-class and the dominant sub-class center vector exceeds a threshold distance.

9. The method of claim 8, wherein, Discarding a non-dominant sub-class from a class is performed according to a threshold condition being met.

10. The method of claim 9, wherein, The threshold condition is a first threshold number of training rounds being exceeded.

11. The method of claim 9, wherein, The method further comprises discarding all non-dominant sub-classes according to another threshold condition being met.

12. The method of claim 11, wherein, The other threshold condition is a second threshold number of training rounds being exceeded.

13. The method of claim 1 or 2, wherein, The image is a human face image.

14. The method of claim 1 or 2, wherein, The class corresponds to a classification condition of the image, and the image is from a batch containing label noise, such that the batch includes at least one image labeled as related to a class that does not correspond to a classification condition of the at least one image.

15. A computer-implemented image classification method, characterized by, The method comprises: inputting a first image to a neural network to map the first image to an embedding vector; identifying, using the neural network, a class corresponding to the first image based on the embedding vector, wherein the neural network has been trained using the method of any of the preceding claims.

16. A system, comprising: The system comprises: one or more processors; and a memory comprising computer-readable instructions that, when executed by the one or more processors, cause the system to perform the method of any of claims 1-15.

17. A computer program product, characterised in that, The computer program product comprises computer-readable instructions that, when executed by a computing device, cause the computing device to perform the method of any of claims 1-15.

Images