A result prediction method based on uncertainty evaluation under label noise
Patent Information
- Application Number
- CN202210529489.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-16
- Publication Date
- 2026-10-20
- Estimated Expiration
- 2042-05-16
AI Technical Summary
[0007]本发明要解决的技术问题在于，针对现有技术的上述缺陷，提供一种针对标签噪声的不确定性评估，旨在解决现有技术中预测不确定性量化方法都不够鲁棒，即使是很少比例的标签噪声，预测不确定性用于误分类检测和OOD数据检测的效果都会有大幅下降的问题
[0038]本发明的有益效果：本发明实施例首先获取真实场景数据；其中，所述真实场景数据中的部分数据与训练数据分布相异；然后将真实场景数据输入至已训练的深度神经网络模型和函数模型，得到第一不确定性预测值和初始预测结果，其中，所述深度神经网络模型基于若干次迭代训练得到，每次迭代训练过程中，对训练数据添加权重，并根据权重对训练数据进行筛选；最后根据所述第一不确定性预测值和所述初始预测结果，得到目标预测结果；可见，本发明实施例通过在训练阶段对训练数据添加权重，并根据权重筛选训练样本，在不影响深度神经网络模型的泛化性的条件下，消除标签噪声对预测不确定性的影响，使得预测结果更加准确。
Smart Images
Figure CN116051880B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a method for predicting results based on uncertainty assessment under labeled noise. Background Technology
[0002] In the field of machine learning, the data used to train models is usually called ID-distribution (ID) data, while data with a different ID distribution than the training data is called Out-of-distribution (OOD) data. In real-world applications, training data is finite and cannot possibly encompass all possible scenarios; therefore, OOD data will appear in real-world applications. For example, an elephant on the road or an overturned truck would typically not appear in the training ID data, but it is possible for them to appear in real-world scenarios.
[0003] In recent years, deep learning, represented by deep neural networks, has achieved remarkable success and has been widely applied in many fields. In tasks such as image classification and speech recognition, neural networks can achieve excellent results when the input data is independently and identically distributed with the training ID data. However, they often fail to recognize when these predictions might be incorrect. More seriously, in the presence of OOD (Out-of-Depth) data, the predictions of neural network models are unreliable. Neural network models cannot provide correct predictions for OOD data, nor do they offer low confidence levels for OOD predictions; instead, they often incorrectly predict it as a label from the ID data with high confidence.
[0004] Deep neural networks perform well in providing accurate predictions, but the lack of understanding of the reliability of a given output is a major obstacle to deploying such models in safety-critical applications. In safety-critical fields such as medical diagnostics and autonomous driving, neural network models need to accurately assess the certainty of their predictions, i.e., quantify prediction uncertainty. Furthermore, neural network models need to assign high confidence to correctly predicted data, low confidence to incorrectly predicted data, and low confidence to unpredictable, out-of-order (OOD) data that is inconsistent with the training data distribution.
[0005] Several methods have recently been proposed to quantify prediction uncertainty in deep neural networks. These methods can be used for misclassification and OOD detection, achieving good results when there is no label noise in the training data. However, in real-world applications, training data contains various labeling errors; even widely used datasets have a significant proportion of label noise. In such scenarios, existing prediction uncertainty quantification methods are not robust enough; even a small proportion of label noise can significantly reduce the effectiveness of prediction uncertainty in misclassification and OOD detection.
[0006] Therefore, existing technologies still need improvement and development. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide an uncertainty assessment for label noise in view of the above-mentioned defects of the prior art. It aims to solve the problem that the existing methods for predicting uncertainty quantification are not robust enough. Even with a small proportion of label noise, the effect of predicting uncertainty on misclassification detection and OOD data detection will be greatly reduced.
[0008] The technical solution adopted by this invention to solve the problem is as follows:
[0009] In a first aspect, embodiments of the present invention provide a method for predicting results based on uncertainty assessment under label noise, wherein the method includes:
[0010] Acquire real-world scene data; wherein, some data in the real-world scene data have a different distribution from the training data.
[0011] Real-world scene data is input into a trained deep neural network model and function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration, weights are added to the training data, and the training data is filtered according to the weights.
[0012] Based on the first uncertainty prediction value and the initial prediction result, the target prediction result is obtained.
[0013] In one implementation, the training process of the deep neural network model includes:
[0014] Obtain an initial training dataset containing label noise; wherein the label noise is noise caused by incorrect labeling of the training data;
[0015] The initial training dataset is input into the initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset. Based on the second uncertainty prediction set, a weight set corresponding to the initial training dataset is obtained. Based on the weight set and the initial training dataset, an intermediate training dataset is obtained. The intermediate training dataset is used as the initial training dataset, and the steps of inputting the initial training dataset into the initial deep neural network model and the function model are repeated.
[0016] When the initial deep neural network model meets the preset training conditions, the iterative training stops, and the trained deep neural network model is obtained.
[0017] In one implementation, the step of inputting the initial training dataset into an initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset, obtaining a weight set corresponding to the initial training dataset based on the second uncertainty prediction set, obtaining an intermediate training dataset based on the weight set and the initial training dataset, using the intermediate training dataset as the initial training dataset, and repeatedly performing the step of inputting the initial training dataset into the initial deep neural network model and the function model includes:
[0018] For the t-th iteration of training, obtain the loss function value set corresponding to the initial training dataset and the second uncertainty prediction value set corresponding to the initial training dataset in the t-th iteration of training. Based on the loss function value set and the second uncertainty prediction value set, obtain the training data weight set corresponding to the initial training dataset in the t-th iteration of training. Based on the training data weight set, weight the initial training dataset in the t-th iteration of training to obtain an intermediate training dataset; wherein, the training data weight set is used to characterize the importance of the training data in the initial deep neural network model; wherein, t is greater than or equal to 1;
[0019] The intermediate training dataset is input into the initial deep neural network model and the function model for the (t+1)th iteration of training.
[0020] In one implementation, the initial training dataset includes several initial training data sets; obtaining the loss function value set corresponding to the initial training dataset in the t-th iteration training and the second uncertainty prediction value set corresponding to the initial training dataset in the t-th iteration training, and obtaining the training data weight set corresponding to the initial training dataset in the t-th iteration training based on the loss function value set and the second uncertainty prediction value set, includes:
[0021] For each set of initial training data, obtain the loss function value corresponding to the initial training data in the t-th iteration of training; input the initial training data into the initial deep neural network model multiple times to obtain several output results, and calculate the variance of the several output results based on the function model to obtain a second uncertainty prediction value corresponding to the initial training data; subtract the loss function value from the obtained preset first value to obtain a difference; multiply the difference by the second uncertainty prediction value to obtain a first product; use the first product as the exponent and the obtained preset second value as the base to obtain the exponentiation result; use the exponentiation result as the training data weight corresponding to the initial training data in the t-th iteration of training.
[0022] The training data weights corresponding to all the initial training data in the initial training dataset are used as the training data weight set corresponding to the initial training dataset.
[0023] In one implementation, the step of weighting the initial training dataset in the t-th iteration of training based on the training data weight set to obtain the intermediate training dataset includes:
[0024] Obtain the preset total number of training iterations and the number of all initial training data in the initial training dataset;
[0025] Multiply t by the number mentioned above to obtain the second product;
[0026] Divide the second product by the total number of training iterations to obtain the quotient.
[0027] Based on the quotient and the training data weight set, an intermediate training dataset is obtained.
[0028] In one implementation, obtaining the intermediate training dataset based on the quotient and the training data weight set includes:
[0029] The reference weight is obtained by finding the weight corresponding to the quotient value in the training data weight set.
[0030] The training data weights that are less than the reference weights in the training data weight set are reset to 0 to obtain the intermediate training dataset.
[0031] In one implementation, obtaining the target prediction result based on the first uncertainty prediction value and the initial prediction result includes:
[0032] When the first uncertainty prediction value is less than a preset threshold, the initial prediction result is taken as the target prediction result.
[0033] Secondly, embodiments of the present invention also provide a result prediction device based on uncertainty assessment under labeled noise, wherein the device includes: a real scene data acquisition module, used to acquire real scene data; wherein, some data in the real scene data has a different distribution from the training data;
[0034] The prediction module is used to input real scene data into a trained deep neural network model and a preset function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration of training, weights are added to the training data and the training data is filtered according to the weights.
[0035] The target prediction result acquisition module is used to obtain the target prediction result based on the first uncertainty prediction value and the initial prediction result.
[0036] Thirdly, embodiments of the present invention also provide a smart terminal, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including a result prediction method based on uncertainty assessment under tag noise as described in any of the above.
[0037] Fourthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform the result prediction method based on uncertainty assessment under tag noise as described in any of the above.
[0038] The beneficial effects of this invention are as follows: First, real-world scene data is acquired; wherein, some data in the real-world scene data has a different distribution from the training data; then, the real-world scene data is input into a trained deep neural network model and a function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is trained based on several iterations. During each iteration, weights are added to the training data, and the training data is filtered according to these weights. Finally, the target prediction result is obtained based on the first uncertainty prediction value and the initial prediction result. Therefore, this invention, by adding weights to the training data during the training phase and filtering training samples according to these weights, eliminates the influence of label noise on prediction uncertainty without affecting the generalization ability of the deep neural network model, resulting in more accurate prediction results. Attached Figure Description
[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0040] Figure 1 This is a schematic diagram of the result prediction method based on uncertainty assessment under tag noise provided in an embodiment of the present invention.
[0041] Figure 2 This is a framework diagram of a result prediction method based on uncertainty assessment under labeled noise, provided as an embodiment of the present invention.
[0042] Figure 3 This is a schematic flowchart of a result prediction method based on uncertainty assessment under label noise, provided as an embodiment of the present invention.
[0043] Figure 4 This is a block diagram illustrating the principle of a result prediction device based on uncertainty assessment under tag noise, as provided in an embodiment of the present invention.
[0044] Figure 5 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation
[0045] This invention discloses a method for predicting results based on uncertainty assessment under labeled noise. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.
[0046] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.
[0047] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0048] In existing technologies, a common approach to assessing the uncertainty of neural network predictions relies on Bayesian methods (e.g., variational Bayesian or Markov chain Monte Carlo), the core of which is calculating the posterior distribution of the neural network parameters. However, exact Bayesian inference is often difficult to process, so only approximate posterior calculations are possible. In practical scenarios, approximate Bayesian inference methods are typically used.
[0049] Monte Carlo Dropout (MC-Dropout) can be interpreted as an ensemble form with shared network parameters, or as an approximate Bayesian inference. Dropout typically involves randomly discarding a certain proportion of neurons during training to regularize and prevent overfitting. MC-Dropout also uses dropout during inference to estimate the prediction distribution. With dropout enabled, the prediction result is not fixed; each forward pass can be viewed as sampling from the posterior distribution of the network weights. The mean and variance of multiple random forward passes represent the prediction result and prediction uncertainty, respectively.
[0050] When the training data contains no label noise, the prediction uncertainty evaluated by MC-Dropout can be effectively used for misclassification and OOD detection. However, when label noise is present in the training data, the performance of MC-Dropout for misclassification and OOD detection drops significantly. Even a small proportion of label noise can have a significant impact on detection performance. Since real-world scene annotations inevitably contain some degree of annotation error and label noise, the use of MC-Dropout is greatly limited.
[0051] To address the problems of existing technologies, this embodiment provides a result prediction method based on uncertainty assessment under label noise. By adding weights to the training data during the training phase and filtering training samples according to the weights, the influence of label noise on prediction uncertainty is eliminated without affecting the generalization of the deep neural network model, resulting in more accurate prediction results. Specifically, real-world scene data is first acquired; some data in the real-world scene data has a different distribution from the training data. Then, the real-world scene data is input into the trained deep neural network model and function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is trained through several iterations. During each iteration, weights are added to the training data, and the training data is filtered according to the weights. Finally, the target prediction result is obtained based on the first uncertainty prediction value and the initial prediction result.
[0052] Exemplary methods
[0053] This embodiment provides a result prediction method based on uncertainty assessment under labeled noise, which can be applied to intelligent terminals for machine learning. Specifically, as follows... Figure 1-2 As shown, the method includes:
[0054] Step S100: Obtain real-world scene data; wherein, some data in the real-world scene data have a different distribution from the training data.
[0055] Specifically, real-world scene data is acquired, and some of the data in the real-world scene data has a different distribution than the training data; that is, the real-world scene data contains Out-of-distribution (OOD) data that is inconsistent with the distribution of the training data ID (Id-distribution).
[0056] Once the real-world scenario data is obtained, actions such as... can be executed. Figure 1 The following steps are shown: S200, Input real scene data into the trained deep neural network model and function model to obtain the first uncertainty prediction value and the initial prediction result, wherein the deep neural network model is obtained based on several iterations of training, and in each iteration of training, weights are added to the training data and the training data is filtered according to the weights;
[0057] Specifically, since the trained deep neural network model is obtained through several iterations of training, weights are added to the training data during each iteration, and the training data is filtered according to the weights. Therefore, the deep neural network model obtained after training will not exhibit generalization and the influence of label noise on the uncertainty of prediction is eliminated. The real scene data is input into the trained deep neural network model, and the deep neural network model outputs an initial prediction result. The real scene data is input into the trained deep neural network model multiple times, and based on the function model (MC-Dropout), the variance of the multiple initial prediction results is calculated to obtain the first uncertainty prediction value.
[0058] In one implementation, the training process of the deep neural network model includes the following steps: obtaining an initial training dataset containing label noise; wherein the label noise is noise caused by incorrect labeling of the training data; inputting the initial training dataset into an initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset; obtaining a weight set corresponding to the initial training dataset based on the second uncertainty prediction set; obtaining an intermediate training dataset based on the weight set and the initial training dataset; using the intermediate training dataset as the initial training dataset, and repeatedly executing the step of inputting the initial training dataset into the initial deep neural network model and the function model; stopping iterative training when the initial deep neural network model meets preset training conditions, and obtaining a trained deep neural network model.
[0059] Specifically, due to the presence of a certain proportion of label noise in the training data, the performance of widely used uncertainty assessment methods based on MC-Dropout is significantly reduced, even to the point of being completely random. The technical problem this invention aims to solve is to develop a robust uncertainty assessment method for training data containing label noise, thereby achieving accurate prediction of results. This invention obtains an initial training dataset containing label noise to prepare for subsequent training.
[0060] In this embodiment, the function model is MC-Dropout. That is, in scenarios where the initial training dataset contains label noise, this invention is based on the MC-Dropout framework. When the initial training dataset is input into the initial deep neural network model, there will be a model output. When the same initial training data is input into the initial deep neural network model multiple times, the variance of multiple model outputs, i.e., the second uncertainty prediction value, can be obtained through MC-Dropout. Therefore, when all initial training datasets are input into the initial deep neural network model, the second uncertainty prediction value set corresponding to the initial training dataset is obtained through MC-Dropout. In order to further eliminate the influence of label noise in the initial training data, a weight set is obtained based on the second uncertainty prediction value set. At the same time, the loss function value set obtained from the network model can be used. Then, mathematical calculations are performed on the loss function value set and the second uncertainty prediction value set to obtain the weight set. Other methods can also be used to obtain the weight set, and there are no specific limitations. With the weight set, the initial training dataset can be weighted according to the weight set to obtain an updated initial training dataset, i.e., an intermediate training dataset. This intermediate training dataset is then used as the initial training dataset and output again into the deep neural network model. The aforementioned steps of inputting the initial training dataset into the initial deep neural network model and the function model to obtain a second set of uncertainty prediction values corresponding to the initial training dataset are repeated. In other words, this invention adds weights to training samples during model training and filters samples based on these weights, thereby eliminating the impact of label noise on prediction uncertainty without affecting the model's generalization ability.
[0061] In this embodiment, when the number of training iterations t of the initial deep neural network model reaches a maximum value, such as T times, the iterative training is stopped, and the trained deep neural network model is obtained, which can then be used in practical application scenarios.
[0062] In one implementation, the step of inputting the initial training dataset into the initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset, obtaining a weight set corresponding to the initial training dataset based on the second uncertainty prediction set, obtaining an intermediate training dataset based on the weight set and the initial training dataset, using the intermediate training dataset as the initial training dataset, and repeatedly performing the step of inputting the initial training dataset into the initial deep neural network model and the function model includes the following steps: for the t-th iteration of training, obtaining the loss function value corresponding to the initial training dataset in the t-th iteration of training. The set of loss function values and the second set of uncertainty prediction values corresponding to the initial training dataset in the t-th iteration of training are used to obtain the training data weight set corresponding to the initial training dataset in the t-th iteration of training. Based on the training data weight set, the initial training dataset in the t-th iteration of training is weighted to obtain an intermediate training dataset. The training data weight set is used to characterize the importance of the training data in the initial deep neural network model. The t is greater than or equal to 1. The intermediate training dataset is input into the initial deep neural network model and the function model for the (t+1)-th iteration of training.
[0063] Specifically, in fields with high security requirements, and where it is impossible to guarantee the complete accuracy of the labels in the training data, in order to ensure the effectiveness of the predictive uncertainty calculated by the neural network model, the following methods can be adopted: Figure 3The flowchart illustrates the solution. During training, different samples will have different loss function values. A small loss function value for a sample indicates that the sample is easy to learn. Samples with large loss function values fall into two categories: data on the decision boundary between different classes and mislabeled data. Both types of data are relatively difficult to learn. It can be seen that samples with large loss function values can be either mislabeled or on the decision boundary. Mislabeled samples are detrimental to model training, while samples on the decision boundary contain more information and are key to model optimization. Samples with small loss function values generally do not contain label noise and should be given higher weights during training. However, if only samples with small loss function values (mostly correctly labeled samples) are used for training, the model's generalization will be poor due to the smaller sample size and the fact that these samples are not on the decision boundary and contain less information. Therefore, when weighting training samples, in addition to considering the sample's loss function value, other factors need to be considered, such as prediction uncertainty. Prediction uncertainty can be used for misclassification detection and OOD data detection. Samples with high prediction uncertainty may lie in two regions: the decision boundary between different categories of data, and the region where OOD (Out-of-Depth) data is located outside the training data distribution. Training data is typically considered ID data; therefore, when training a neural network model, each training sample point is considered to be in the region of ID data, not the region of OOD data. Thus, training samples with high prediction uncertainty can only be samples located on the decision boundary. These samples contain more information and require higher weights. Furthermore, different training stages also affect the elimination of label noise: although weighting training samples using loss function values and prediction uncertainty can significantly reduce the impact of mislabeled data on the generalization ability of the neural network model, even giving small weights to mislabeled data can still affect the accuracy of prediction uncertainty. Simultaneously, mislabeled data helps the neural network model learn data features and improve generalization in the early stages of training, but in the later stages, the deep neural network model will remember the incorrect labels, thus reducing performance. Therefore, the weights of training samples need to be dynamically adjusted according to the training stage. As training progresses, the weights of samples that are more likely to be mislabeled should be set to 0, i.e., training should be filtered based on weights. Considering the impact of the three factors mentioned above (loss function value of training samples, prediction uncertainty, and training stage), in this embodiment, for the t-th iteration of training, that is, in the t-th epoch, the dataset obtained in the t-th iteration of training is compared with the initial training dataset (x). i ,y i (where x) i For the initial training data, y i Let i be the set of loss function values corresponding to the labels of the initial training data (where i is the i-th initial training data). and the second uncertainty prediction set {Uncertainty(x)} corresponding to the initial training dataset in the t-th iteration of training. i Based on the loss function value set and the second uncertainty prediction value set, the training data weight set corresponding to the initial training dataset in the t-th iteration of training is obtained. Based on the training data weight set {w} i}, can be further improved by adjusting the training data weight set {w i The initial training dataset is then filtered to obtain the final training data weight set. Next, the initial training dataset from the t-th iteration is weighted to obtain an intermediate training dataset. This updated initial training dataset (intermediate training dataset) is then input into the initial deep neural network model and the function model for the (t+1)-th iteration. In this way, through multiple iterations, the initial training dataset is continuously filtered, ensuring that the neural network maintains sufficient generalization while removing samples that are more likely to be mislabeled, thus improving the effectiveness in predicting uncertainty.
[0064] In one implementation, for the t-th iteration of training, the initial training dataset includes several initial training data sets; obtaining the loss function value set corresponding to the initial training dataset in the t-th iteration of training and the second uncertainty prediction value set corresponding to the initial training dataset in the t-th iteration of training, and obtaining the training data weight set corresponding to the initial training dataset in the t-th iteration of training based on the loss function value set and the second uncertainty prediction value set, includes the following steps: for each initial training data set, obtaining the loss function value corresponding to the initial training data in the t-th iteration of training; inputting the initial training data multiple times into the initial deep neural network. The model generates several output results, and based on the function model, variance is calculated on the several output results to obtain a second uncertainty prediction value corresponding to the initial training data; the obtained preset first value is subtracted from the loss function value to obtain a difference; the difference is multiplied by the second uncertainty prediction value to obtain a first product; the first product is used as the exponent, and the obtained preset second value is used as the base to obtain the exponentiation result; the exponentiation result is used as the training data weight corresponding to the initial training data in the t-th iteration of training; and all the training data weights corresponding to the initial training data in the initial training dataset are used as the training data weight set corresponding to the initial training dataset.
[0065] Specifically, for each initial training data point in the initial training dataset, the loss function value of the initial training data is first calculated. Among them, y i The labels for the initial training data. The output of the deep neural network is used as the basis for determining the second uncertainty prediction value (Uncertainty(x)) of the initial training data. Then, for an initial training dataset, the initial training data is input into the initial deep neural network model multiple times to obtain several output results. Based on the function model, the variance of these output results is calculated to obtain the second uncertainty prediction value of the initial training data. i Assuming the first value is preset to 0 (other values can be used in practice, which are not restricted here), subtract the loss function value from 0 to obtain the difference. The difference - Multiply by the second uncertainty prediction value Uncertainty(x) i ), to obtain the first product - Assuming the second value is preset to 10, denoted by e, and taking e as the base, subtract the first product... As an index, the weights of the training data are obtained. Thus, the training data weights corresponding to all the initial training data in the initial training dataset are used as the training data weight set {w} corresponding to the initial training dataset. i}
[0066] In one implementation, for the t-th iteration of training, the step of weighting the initial training dataset in the t-th iteration based on the training data weight set to obtain an intermediate training dataset includes the following steps: obtaining a preset total number of iterations and the number of all initial training data in the initial training dataset; multiplying t by the number of data to obtain a second product; dividing the second product by the total number of iterations to obtain a quotient; finding the weight corresponding to the quotient in the training data weight set to obtain a reference weight; and resetting the weights of training data in the training data weight set that are less than the reference weight to 0 to obtain the intermediate training dataset.
[0067] Specifically, assigning smaller weights to mislabeled samples can eliminate the impact of label noise on the generalization of the neural network model, but its impact on prediction uncertainty remains significant. As training progresses, further removal of mislabeled samples is necessary. In this embodiment, the preset total number of training iterations T and the number N of all initial training data in the initial training dataset are obtained. T is multiplied by the number N to obtain a second product t*N. This second product is then divided by the total number of training iterations to obtain the quotient. Find the quotient value in the training data weight set The corresponding weights are used to obtain the reference weight w, and the training data weight set {w} is used to... iThe weights of training data in the dataset smaller than the reference weight w are reset to 0, resulting in an intermediate training dataset. Thus, the larger the training epoch, the more samples have their weights set to 0. Through this iterative process of filtering training samples, the neural network can maintain sufficiently good generalization while removing samples that are more likely to be mislabeled, improving its ability to address prediction uncertainty.
[0068] After obtaining the first uncertainty prediction value and the initial prediction result, the following can be executed: Figure 1-2 The following step S300 shows obtaining the target prediction result based on the first uncertainty prediction value and the initial prediction result.
[0069] Specifically, preset conditions can be set to output the initial prediction result corresponding to the first uncertainty prediction value that meets the preset conditions, thus obtaining the target prediction result. In other words, when this trained deep neural network model is deployed to a real-world scenario, it can provide both the initial prediction result and the first uncertainty prediction value. This first uncertainty prediction value can be used for misclassification detection and OOD data detection. Whether the initial prediction result is ultimately output depends on the first uncertainty prediction value.
[0070] In one implementation, step S300 includes the following steps:
[0071] S301. When the first uncertainty prediction value is less than a preset threshold, the initial prediction result is taken as the target prediction result.
[0072] Specifically, when the first uncertainty prediction value is less than a preset threshold, that is, when the uncertainty is low, the initial prediction result is taken as the target prediction result; when the first uncertainty prediction value is greater than or equal to the preset threshold, that is, when the uncertainty is high, the prediction is rejected and no target prediction result is given.
[0073] Highlights of this invention:
[0074] 1. Dynamically change the weights of samples during training and select training samples based on the weights to eliminate the impact of label noise on prediction uncertainty.
[0075] 2. Give higher weights to samples with lower loss function values.
[0076] 3. Give higher weights to samples with high prediction uncertainty calculated based on MC-Dropout.
[0077] 4. As training progresses, more and more low-weight samples are removed to reduce the impact of label noise on prediction uncertainty.
[0078] Exemplary device
[0079] like Figure 4 As shown in the figure, this embodiment of the invention provides a result prediction device based on uncertainty assessment under label noise. The device includes a real scene data acquisition module 401, a prediction module 402, and a target prediction result acquisition module 403, wherein: the real scene data acquisition module 401 is used to acquire real scene data; wherein, some data in the real scene data has a different distribution from the training data.
[0080] The prediction module 402 is used to input real scene data into a trained deep neural network model and a preset function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration of training, weights are added to the training data and the training data is filtered according to the weights.
[0081] The target prediction result acquisition module 403 is used to obtain the target prediction result based on the first uncertainty prediction value and the initial prediction result.
[0082] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 5 As shown, the smart terminal includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a result prediction method based on uncertainty assessment under tag noise. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the smart terminal to detect the operating temperature of internal devices.
[0083] Those skilled in the art will understand that Figure 5 The schematic diagram in the figure is merely a block diagram of a part of the structure related to the present invention and does not constitute a limitation on the smart terminal on which the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0084] In one embodiment, a smart terminal is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:
[0085] Acquire real-world scene data; wherein, some data in the real-world scene data have a different distribution from the training data.
[0086] Real-world scene data is input into a trained deep neural network model and function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration, weights are added to the training data, and the training data is filtered according to the weights.
[0087] Based on the first uncertainty prediction value and the initial prediction result, the target prediction result is obtained.
[0088] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0089] In summary, this invention discloses a result prediction method based on uncertainty assessment under label noise. The method includes: acquiring real-world scene data; wherein a portion of the data in the real-world scene data has a different distribution from the training data; inputting the real-world scene data into a trained deep neural network model and a function model to obtain a first uncertainty prediction value and an initial prediction result, wherein the deep neural network model is trained based on several iterations, and in each iteration, weights are added to the training data, and the training data is filtered according to the weights; and obtaining a target prediction result based on the first uncertainty prediction value and the initial prediction result. When label noise exists in the training data, existing prediction uncertainty assessment methods are not effective enough. Giving higher weights to samples with low loss function values can retain samples with correct labels, helping to eliminate the influence of label noise. Giving higher weights to samples with high prediction uncertainty can retain samples on the decision boundary, helping to maintain the model's generalization ability. As training progresses, more and more low-weight samples are gradually removed to ensure that the reliability of prediction uncertainty is not affected by incorrect labels. Using these methods to filter training samples allows prediction uncertainty to remain robust even under label noise conditions. Even if there are labeling errors in the training data, the model trained using the method of this invention, when deployed in a real-world scenario, can still provide reliable predictions despite uncertainties. For new samples, the neural network model will provide a prediction result, along with the uncertainty of that prediction. If the uncertainty is high, caution is required in making a judgment, or even a prediction should be rejected. If the uncertainty is low, a prediction result can be provided.
[0090] Based on the above embodiments, the present invention discloses a result prediction method based on uncertainty assessment under label noise. It should be understood that the application of the present invention is not limited to the above examples. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for predicting results based on uncertainty assessment under label noise, characterized in that, The method includes: Acquire real-world scene data; wherein, the real-world scene data includes images, and some data in the real-world scene data have a different distribution from the training data; Real-world scene data is input into a trained deep neural network model and function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration, weights are added to the training data, and the training data is filtered according to the weights and the prediction uncertainty. Among them, training samples with prediction uncertainty greater than a threshold are samples at the decision boundary, and these samples have higher weights than training samples with prediction uncertainty less than a threshold. During the training process of the deep neural network model, the weights of the training samples are dynamically adjusted according to the training stage. Samples with loss function values less than a first preset value and samples with prediction uncertainty greater than a second preset value are assigned weights greater than a third prediction value. For the t-th iteration of training, obtain the preset total number of training iterations and the number of all initial training data in the initial training dataset; For each initial training data, obtain the loss function value corresponding to the initial training data in the t-th iteration of training; input the initial training data into the initial deep neural network model multiple times to obtain several output results, and calculate the variance of the several output results based on the function model to obtain a second uncertainty prediction value corresponding to the initial training data; subtract the loss function value from the obtained preset first value to obtain the difference; multiply the difference by the second uncertainty prediction value to obtain a first product; use the first product as the exponent and the obtained preset second value as the base to obtain the exponentiation result; use the exponentiation result as the training data weight corresponding to the initial training data in the t-th iteration of training. The training data weights corresponding to all the initial training data in the initial training dataset are used as the training data weight set corresponding to the initial training dataset. Multiply t by the number mentioned above to obtain the second product; Divide the second product by the total number of training iterations to obtain the quotient. The reference weight is obtained by finding the weight corresponding to the quotient value in the training data weight set; The weights of the training data whose weights are less than the reference weights are reset to 0 to obtain an intermediate training dataset, which is then used as the initial training dataset for the next iteration. Based on the first uncertainty prediction value and the initial prediction result, the target prediction result is obtained; The step of obtaining the target prediction result based on the first uncertainty prediction value and the initial prediction result includes: When the first uncertainty prediction value is less than a preset threshold, the initial prediction result is taken as the target prediction result.
2. The result prediction method based on uncertainty assessment under tag noise according to claim 1, characterized in that, The training process of the deep neural network model includes: Obtain an initial training dataset containing label noise; wherein the label noise is noise caused by incorrect labeling of the training data; The initial training dataset is input into the initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset. Based on the second uncertainty prediction set, a weight set corresponding to the initial training dataset is obtained. Based on the weight set and the initial training dataset, an intermediate training dataset is obtained. The intermediate training dataset is used as the initial training dataset, and the steps of inputting the initial training dataset into the initial deep neural network model and the function model are repeated. When the initial deep neural network model meets the preset training conditions, the iterative training stops, and the trained deep neural network model is obtained.
3. The result prediction method based on uncertainty assessment under label noise according to claim 2, characterized in that, The steps of inputting the initial training dataset into the initial deep neural network model and the function model to obtain a second uncertainty prediction set corresponding to the initial training dataset, obtaining a weight set corresponding to the initial training dataset based on the second uncertainty prediction set, obtaining an intermediate training dataset based on the weight set and the initial training dataset, using the intermediate training dataset as the initial training dataset, and repeatedly performing the step of inputting the initial training dataset into the initial deep neural network model and the function model include: For the t-th iteration of training, obtain the loss function value set corresponding to the initial training dataset and the second uncertainty prediction value set corresponding to the initial training dataset in the t-th iteration of training. Based on the loss function value set and the second uncertainty prediction value set, obtain the training data weight set corresponding to the initial training dataset in the t-th iteration of training. Based on the training data weight set, weight the initial training dataset in the t-th iteration of training to obtain an intermediate training dataset; wherein, the training data weight set is used to characterize the importance of the training data in the initial deep neural network model; wherein, t is greater than or equal to 1; The intermediate training dataset is input into the initial deep neural network model and the function model for the (t+1)th iteration of training.
4. A result prediction device based on uncertainty assessment under label noise, characterized in that, The result prediction device based on uncertainty assessment under label noise is applied to the result prediction method based on uncertainty assessment under label noise as described in any one of claims 1-3, and the device includes: A real-scene data acquisition module is used to acquire real-scene data; wherein, the real-scene data includes images, and some data in the real-scene data have a different distribution from the training data; The prediction module is used to input real scene data into a trained deep neural network model and a preset function model to obtain a first uncertainty prediction value and an initial prediction result. The deep neural network model is obtained based on several iterations of training. During each iteration of training, weights are added to the training data and the training data is filtered according to the weights. The target prediction result acquisition module is used to obtain the target prediction result based on the first uncertainty prediction value and the initial prediction result.
5. A smart terminal, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include methods for performing any one of claims 1-3.
6. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-3.