A method for training a deep neural network and such a network

Abstract

A method for training a deep neural network using a labeled dataset, wherein at least two models are trained collaboratively and concurrently, and wherein each model is trained with a supervised learning loss and an imitation loss, wherein the supervised learning loss involves learning from environmental cues and the supervision from the imitation loss is related to imitation in culture leaning.

Description

Technical Field

[0001] The present invention relates to a method for training a deep neural network using a data set with annotated labels. Background Art

[0002] Deep neural networks (DNNs) have been shown to easily adapt to random labels. [2] , which makes it challenging to train models effectively. The existing technical methods for training with noisy labels can be roughly divided into two methods: i) Correcting labels by estimating the noise transition matrix [3，7] , ii) Identify noisy labels to filter out [4，8] These samples may reduce their weight [5，6] However, the former method relies on accurately estimating the noise transition matrix, which is difficult, especially when there are many categories; the latter method requires an effective method for identifying noisy labels and / or estimating the percentage of noisy samples. Among these methods, more emphasis is currently placed on separating noisy samples from clean samples, where the common criterion is to regard low-loss samples as representatives of clean labels. [1，4] However, harder examples may be considered noisy, so the model may be biased towards easy instances. Both methods consider annotation quality as the main reason for the degradation of model performance, so the proposed solutions rely on accurately re-annotating, filtering out or down-weighting examples with incorrect labels. Summary of the invention

[0003] In contrast to previous approaches, the present invention focuses on making the underlying training framework more robust to noisy labels, rather than focusing on annotation. The lack of robustness of standard training procedures can be attributed to many factors. The cross entropy loss maximizes the mutual information boundary between the one-hot encoded labels and the learned representation. The trained model does not obtain information about the similarity of data points between categories, and thus lacks a source of useful information about the samples when the labels provided are incorrect, or lacks additional supervision to mitigate the adverse effects of noisy labels. Moreover, there is a lack of regularization means to prevent the model from memorizing the training labels.

[0004] In order to at least partially address the above-mentioned shortcomings in neural network training, according to the method of the present invention, at least two models are collaboratively trained simultaneously, wherein each model is trained with a supervised learning loss and an imitation loss, wherein the supervised learning loss involves learning from a ground-truth label, and the supervision from the imitation loss involves alignment of the outputs of the two models.

[0005] Therefore, in addition to the supervised learning loss, each model is trained with an imitation loss that aligns the posterior distributions of the two models to build a consensus on the secondary class probabilities and primary class predictions. The method of the present invention is called Noisy Concurrent Training (NCT).

[0006] Advantageously, the two models are initialized separately.

[0007] Specifically, NCT involves training models collaboratively, whereby each model is trained through a combination of supervised learning loss and imitation loss. Although the ground truth labels (environmental factors) may be noisy, DNNs tend to prioritize learning simple patterns before memorizing noisy labels, so in the initial stage of learning, model training focuses on using supervised learning loss and gradually increasing the fitness of the two models (populations).

[0008] After the initial phase of learning comes the phase of training advancement, where the focus of model training shifts to reliance on imitation loss, where the relative weight of supervised learning loss decreases. As training advances, the information quality threshold therefore increases, so that models can rely more on imitating each other and building consensus. This is simulated using a dynamic balancing scheme that gradually increases the weight of imitation loss while reducing the weight of supervised learning loss. Therefore, as training advances, the models build consensus on their accumulated knowledge and align their posterior probability distributions. To train the model, the imitation loss provides an additional supervisory signal in addition to the one-hot labels. In this way, the model can also learn useful information from training samples with incorrect labels.

[0009] Furthermore, to prevent memorization, it is preferred that during the training process, the labels of random samples drawn in batches from the dataset are changed to random classes drawn independently by at least two models from a uniform distribution over the total number of classes in each batch. This technique, called target variability, has multiple uses: it signals the model not to rely too much on noisy labels, thereby implicitly raising the information quality threshold, strongly inhibits memorization of training labels, and also keeps the two models sufficiently differentiated to avoid confirmation bias caused by the method degenerating into self-training.

[0010] Preferably, target variability is applied to each model independently so that the two networks remain sufficiently differentiated that different types of errors can be filtered jointly.

[0011] Advantageously, the target variability rate is initially low to allow the model to effectively learn simple patterns, and is gradually increased during training to combat the model's tendency to memorize.

[0012] The method of the present invention obtains a robust learning framework that allows deep neural networks to be effectively trained under conditions where the label has a high level of noise. This significantly improves the practicality of the model in practical situations where the annotation quality is usually not ideal.

[0013] Therefore, the method of the present invention enables the use of large-scale automatically annotated and crowdsourced datasets to learn rich representations, which can be used for subsequent downstream tasks such as segmentation, detection, and depth estimation. The improved representation can improve the performance of downstream tasks and has wide applications in various industries such as self-driving cars and / or high-precision map creation.

[0014] The deep neural network according to the present invention is preferably used as a framework for one or more subsequent image or video tasks selected from the group consisting of segmentation, detection and depth estimation.

[0015] In addition, the deep neural network according to the present invention is preferably embedded in an autonomous driving and / or high-precision map updating system.

[0016] In the following, the invention will be further explained with reference to an exemplary embodiment of the method according to the invention, without the invention being restricted to this exemplary embodiment but rather to the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the accompanying drawings, a single Figure 1 The schematic diagram shows that two models are trained collaboratively at the same time. DETAILED DESCRIPTION

[0018] Given a dataset D = {x(i), y(i)} (i = 1 to N) containing N samples, where x(i) is the input image and y(i) is the one-hot ground truth label on C classes that may be noisy, the proposed method NCT is formulated as a dynamic collaborative learning between a cohort of two networks parameterized by θ1 and θ2. Each network is trained with a supervised loss (standard cross entropy L CE ) and the imitation loss (Kullback-Leibler divergence, D KL ) for training. The total loss of each model is as follows:

[0019]

[0020] where σ is the softmax function, z θ is the output logit, and τ is the temperature, which is usually set to 1. Using higher values ​​of τ will produce a softer probability distribution over the classes. Adjusting the parameter α∈[0,1] controls the relative weight between the two losses mentioned above.

[0021] For inference, the average ensemble of the two models was used.

[0022]

[0023] dynamic equilibrium

[0024] Given a mixture of clean labels and noisy labels, DNNs tend to prioritize learning simple patterns and fitting clean data before memorizing noisy labels [2]. NCT adopts a dynamic balancing scheme, so that the initial two networks learn more from the supervised loss, that is, they have a smaller α d values, and as training progresses, these networks focus more on building consensus and KL Adjust its posterior distribution, i.e. α d →1. To simulate this behavior, the following sigmoid ramp-up function

[10] is used.

[0025]

[0026] where α max is the maximum α value, e is the current epoch, and e r is the length of the slope growth (i.e., α d epoch at which the maximum value is reached), and β controls the shape of the function. Figure 1 The dynamic equilibrium functions for different values ​​of β are shown.

[0027] Dynamic target variability

[0028] NCT uses target variability, whereby for each sample in the training batch, the one-hot label is changed to a random class sampled from a uniform distribution over categories C with probability r. Target variability acts as a regularizer, preventing the models from memorizing labels. Target variability is applied to each model independently, so that the two networks remain sufficiently differentiated so that different types of errors can be jointly filtered. Since networks tend to memorize noisy labels in the later stages of training, NCT employs dynamic target variability, so that the target variability rate is low in the initial epoch and gradually increases during training ( Figure 1 ). NCT uses a logarithmic ramp growth function,

[0029]

[0030] where r min and r max are the minimum and maximum target variable rates, e is the current epoch, and e max is the total number of epochs, e w is the preheating length. The details of this method are summarized in Algorithms 1 and 2.

[0031]

[0032]

[0033] result

[0034] Next, NCT is compared to multiple baseline methods under similar experimental settings. Since the quality of the dataset is not known a priori, the learning method should be applicable to both noisy and clean datasets. Therefore, we compare methods on clean and different levels of label noise. Table 1 shows consistent improvements for lower noise levels. On the clean CIFAR-100, the gap between M-Correction and NCT is considerable. However, compared to M-Correction, the proposed method is not optimal for very high levels of symmetric noise (50%). Table 2 shows that the effectiveness of the proposed method is extended beyond the CIFAR dataset to the complex Tiny-ImageNet classification task. For symmetric noise, similar patterns as the CIFAR dataset emerge. For asymmetric noise, which may better simulate actual noise, NCT provides significant improvements in generalization. M-Correction exhibits unstable behavior on asymmetric noise, which is manifested as a high standard deviation in performance.

[0035] To verify the practical application of NCT, the method is further compared on two real noisy datasets. Table 3 shows that NCT provides considerable performance improvement (top1 accuracy is improved by about 10%) compared with the state-of-the-art methods on the WebVision dataset. For Clothing1M, Table 4 provides the marginal gain over P-type correction.

[0036] Empirical results on clean and noisy versions of benchmark datasets, as well as consistent improvements on real-world noisy datasets, demonstrate the effectiveness of NCT as a general learning framework that is robust to label noise.

[0037] Table 1. Comparison with existing methods on CIFAR-10 and CIFAR-100 datasets with symmetric noise. The baseline results are reproduced from Arazo et al. [1], compared with which our method shows the highest test accuracy (%) in all epochs (best) and final epoch accuracy (worst). For our method, we report the mean and 1STD for three different seed values.

[0038]

[0039] Table 2. Comparison with existing methods on the Tiny-ImageNet dataset with symmetric and asymmetric pair flip noise. The baseline results are reproduced from Yu et al. [8], and the proposed method shows the best (Best) and average (Avg.) test accuracy (%) over the last 10 epochs. For a fair comparison, M-Correction is run on noisy simulations using the public code and hyperparameters mentioned in their paper [8]. We also run Standard and Co-teaching+ on the clean dataset. For all these experiments performed, we report the average and 1STD for three different seed values.

[0040]

[0041] Table 3. Comparison with previous methods trained on the WebVision dataset. The baseline results are reproduced from Chen et al.

[14] , and we report the final accuracy (%) on the WebVision and ImageNet ILSVRC12 validation sets for comparison. For our method, we report the mean and 1STD for three different seed values.

[0042]

[0043] Table 4. Comparison with existing methods on Clothing1M. The baseline results are copied from the original paper, and we report the best test accuracy (%) compared to them. For our method, we report the average and 1STD for three different seed values.

[0044]

[0045] Table 5. Target variable rate parameter r max Impact on CIFAR-10. We report the highest test accuracy (%) over all epochs (best) and the accuracy at the final epoch (worst). The average and 1STD for three different seed values ​​are reported.

[0046]

[0047] The impact of target variability

[0048] In order to analyze the sensitivity of the proposed method to the target variability parameters, the CIFAR-10 dataset was used, and the experimental settings were the same as the above experiments. The experiment shows that changing r while keeping all other parameters unchanged max Table 5 shows that, compared with the no-target variability (r max =0), target variability provides a significant performance improvement compared to the baseline NCT method. max Wide target variability range of ≤0.7, NCT for r max The method is less sensitive to the choice of the value of r for a higher noise level (50%) than for a lower noise level (20%). max The value is more sensitive.

[0049] Although the present invention has been discussed above with reference to an exemplary embodiment of the method of the present invention, the present invention is not limited to this particular embodiment, which may be varied in many ways without departing from the present invention. Therefore, the appended claims should not be interpreted strictly in accordance with the exemplary embodiment discussed. On the contrary, this embodiment is intended only to explain the wording of the appended claims, and is not intended to limit the claims to this exemplary embodiment. Therefore, the scope of protection of the present invention should be interpreted only in accordance with the appended claims, wherein the use of this exemplary embodiment can resolve possible ambiguities in the wording of the claims.

[0050] References

[0051] [1] Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. arXivpreprint arXiv:1904.11238, 2019.

[0052] [2] Devansh Arpit, Nicolas Ballas,David Krueger,Emmanuel Bengio,Maxinder S Kanwal,Tegan Maharaj,Asja Fischer,AaronCourville,Yoshua Bengio,et al.A closer look at memorization in deep networks.InProceedings of the 34th International Conference on Machine Learning-Volume70,pages 233–242.JMLR.org,2017.

[0053] [3]Jacob Goldberger and Ehud Ben-Reuven.Training deep neural-networksusing a noise adaptation layer.2016.

[0054] [4]Bo Han,Quanming Yao,Xingrui Yu,Gang Niu,Miao Xu,Weihua Hu,IvorTsang,and Masashi Sugiyama.Coteaching:Robust training of deep neuralnetworkswith extremely noisy labels.In Advances in neural informationprocessingsystems,pages 8527–8537,2018.

[0055] [5]Lu Jiang,Zhengyuan Zhou,Thomas Leung,Li-Jia Li,and Li Fei-Fei.Mentornet:Learning data-driven curriculum for very deep neural networksoncorrupted labels.arXiv preprint arXiv:1712.05055,2017.

[0056] [6]Eran Malach and Shai Shalev-Shwartz.Decoupling"when to update"from"how to update".In Advances in Neural Information Processing Systems,pages 960–970,2017.

[0057] [7]Giorgio Patrini,Alessandro Rozza,Aditya Menon,Richard Nock,andLizhen Qu.Making neural networks robust to label noise:a losscorrectionapproach.stat,1050:13,2016.

[0058] [8]Xingrui Yu,Bo Han,Jiangchao Yao,Gang Niu,Ivor W Tsang,andMasashiSugiyama.How does disagreement help generalization against labelcorruption?arXiv preprint arXiv:1901.04215,2019.

[0059] [9]Robert Boyd,Peter J Richerson,and Joseph Henrich.The culturalniche:Why social learning is essential for human adaptation.Proceedings ofthe NationalAcademy of Sciences,108(Supplement 2):10918–10925,2011

[0060]

[10] Samuli Laine and Timo Aila.Temporal ensembling for semi-supervisedlearning.arXiv preprint arXiv:1610.02242,2016.

[0061]

[11] Hongyi Zhang,Moustapha Cisse,Yann N Dauphin,and DavidLopez-Paz.mixup:Beyond empirical risk minimization.arXiv preprintarXiv:1710.09412,2017.

[0062]

[12] Scott Reed,Honglak Lee,Dragomir Anguelov,Christian Szegedy,Dumitru Erhan,and Andrew Rabinovich.Train-ing deep neural networks onnoisylabels with bootstrapping.arXiv pre-print arXiv:1412.6596,2014.

[0063]

[13] Xingjun Ma,Yisen Wang,Michael E Houle,Shuo Zhou,Sarah M Erfani,Shu-Tao Xia,Sudanthi Wijewickrema,and James Bai-ley.Dimensionality-drivenlearning with noisy la-bels.arXiv preprint arXiv:1806.02612,2018.

[0064]

[14] Pengfei Chen,Benben Liao,Guangyong Chen,and ShengyuZhang.Understanding and utilizing deep neural networks trained with noisylabels.arXivpreprint arXiv:1905.05040,2019.

[0065]

[15] Daiki Tanaka,Daiki Ikami,Toshihiko Yamasaki,and Kiy-oharuAizawa.Joint optimization framework for learning with noisy labels.InProceedings of theIEEE Conference on Comput-er Vision and PatternRecognition,pages 5552–5560,2018.

[0066]

[16] Weihe Zhang,Yali Wang,and Yu Qiao.Metacleaner:Learn-ingtohallucinate clean representations for noisy-labeled visualrecognition.InProceedings of the IEEE Conference on Computer Vision andPatternRecognition,pages 7373–7382,2019.

[0067]

[17] Junnan Li,Yongkang Wong,Qi Zhao,and Mohan S Kankanha-li.Learningto learn from noisy labeled data.In Proceedings of the IEEE ConferenceonComputer Vision and Pattern Recogni-tion,pages 5051–5059,2019.

[0068]

[18] Kun Yi and Jianxin Wu.Probabilistic end-to-end noise correctionforlearning with noisy labels.In Proceedings of the IEEE Conference onComputerVision and Pattern Recogni-tion,pages 7017–7025,2019.

[0069]

[20] Ying Zhang,Tao Xiang,Timothy M Hospedales,and HuchuanLu.Deepmutual learning.In Proceedings of the IEEE Confer-ence on ComputerVision andPattern Recognition,pages 4320–4328,2018.

[0070]

[21] Geoffrey Hinton,Oriol Vinyals,and Jeff Dean.Dis-tilling theknowledgein a neural network.arXiv preprint arXiv:1503.02531,2015.

Claims

1. A method for training a deep neural network using a labeled dataset, wherein two models are trained collaboratively and concurrently, wherein each model is trained with a supervised learning loss and an imitation loss, wherein the supervised learning loss involves learning from ground truth labels, and the supervision from the imitation loss involves aligning the outputs of the two models; the dataset comprises a plurality of sample images and labels corresponding to each of the sample images; wherein in the initial stage of learning, the focus of the training of the two models is to use the supervised learning loss, thereby gradually increasing the fitness of the two models; in the stage after the initial stage of learning, as the training progresses, the focus of training the two models turns to relying on the imitation loss, wherein the relative weight percentage of the supervised learning loss decreases, while the relative weight of the imitation loss increases; in, The total loss for each model is as follows: Wherein, θ1 is the parameter of one of the two models; is the total loss function of the one model; θ2 is the parameter of the other model in the two models; is the total loss function of the other model; and is the supervision loss of each of the two models; and is the imitation loss of each of the two models; σ is the softmax function; z θ is the output logarithmic probability; τ is the temperature; α is the adjustment parameter, the value range of α is α∈[0,1], and the change of α is described by the Sigmoid ramp growth function: where α max is the maximum α value, e is the current epoch, and e r is the length of the ramp growth, and β is used to control the shape of the Sigmoid ramp growth function. The method according to claim 1 , wherein the two models are initialized separately.

3. A method according to claim 1 or 2, wherein in the stage of training advancement, the two models establish a consensus on their accumulated knowledge while increasing the reliance on the imitation loss and reducing the reliance on adapting the ground truth labels through the supervised loss and aligning the posterior probability distributions of the two models.

4. A method according to claim 1 or 2, wherein target variability is used, wherein during training, the labels of random samples drawn in batches from the dataset are changed to random classes, which are independently drawn by the two models from a uniform distribution of the total number of classes in each batch to suppress the two models' memory of noisy training labels while maintaining the differences between the two models.

5. The method of claim 4, wherein the target variability is applied to each model independently so that the two networks remain sufficiently differentiated so that they can jointly filter different types of errors.

6. The method of claim 4, wherein the target variability is initially low to allow the model to effectively learn simple patterns, and is gradually increased during training to counter the model's tendency to memorize.

7. The method of claim 1, wherein the deep neural network is used as a framework for performing one or more subsequent image or video tasks selected from the group consisting of segmentation, detection, and depth estimation.

8. The method according to claim 1, wherein the deep neural network is implemented in autonomous driving and / or high-precision map production.

Images