DATA AUGMENTATION EVALUATION AND AUTOMATIC TRAINING SCHEME IMPROVEMENT THROUGH TYPIZATION

Abstract

The presented embodiments include methods for evaluating augmented training elements. These augmented training elements can be generated using various augmentation techniques. The disclosed embodiments can generate training sets that are useful for training a wide variety of machine learning models.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, the disclosed embodiments relate to methods for data augmentation (also referred to as data augmentation) useful for generating training sets for machine learning models. BACKGROUND OF THE INVENTION

[0002] Training and developing deep learning models often requires a large number of diverse and informative training samples in training sets that are useful for training the deep learning models. However, collecting high-quality samples can be costly (e.g., in terms of time, labor, and / or computational power) for many applications, such as industrial inspections, automated optical inspections, and medical imaging applications. In practice, data-intensive deep learning methods can suffer from data scarcity. To alleviate data scarcity, data augmentation techniques are sometimes used, which have proven effective for a wide variety of tasks.

[0003] However, not all data augmentations are equally useful. For example, a model trained on many highly noisy images may produce biased predictions in real-world scenarios, as the highly noisy images may differ from the real images (i.e., be far removed from the real ones in typical deep learning embeddings). Previous work has conducted extensive empirical evaluations to determine which data augmentation should be used for which data domain and task. Evaluation is typically performed by comparing the quality of a specific model trained with different data augmentations. SUMMARY

[0004] Repeating training to evaluate each data augmentation type is time-consuming and can waste computational resources. Furthermore, the evaluation results may be biased toward the desired task. For example, image classification tasks and semantic segmentation tasks prefer different data augmentations. Consequently, there is a need in industry for efficient methods for evaluating data augmentations that are suitable for inclusion in a training set useful for training one or more machine learning models.

[0005] The embodiments presented here differ from the prior art solutions in at least two respects. First, the disclosed embodiments require only a single trained generative model for evaluating a multitude of data augmentations, in contrast to the prior art methods, which require a different trained model for evaluating each different augmentation. Second, the disclosed embodiments can use statistics estimated via a generative model, rather than model performance on a specific task.

[0006] Disclosed embodiments include methods comprising: estimating an empirical entropy of a training set on which a generative model was trained; generating a typing score for the distance between 1) an augmented training item of a training item included in the training set and 2) a typing set of the trained generative model, the typing score based on the estimated empirical entropy;and comparing the typing score to a threshold to determine whether the augmented training item is suitable for inclusion in the training set. Other disclosed embodiments include a non-transitory memory containing processor-executable instructions that, when executed by one or more processors, cause a system to perform operations to: estimate an empirical entropy of a training set on which a generative model was trained; generate a typing score for the distance between 1) an augmented training item of a training item included in the training set and 2) a typing set of the trained generative model, the typing score based on the estimated empirical entropy;and comparing the typing score to a threshold to determine whether the augmented training item is suitable for inclusion in the training set. Other disclosed embodiments include systems comprising: one or more processors; and non-transitory memory containing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations including estimating an empirical entropy of a training set on which a generative model was trained; generating a typing score based on the estimated empirical entropy for each of a plurality of augmented training items;Comparing the generated typing score with a threshold for each of the plurality of augmented training items to determine whether the augmented training item is suitable for inclusion in the training set; and including each of the plurality of augmented training items determined to be suitable for inclusion in the training set in the training set. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a flowchart of an example method in accordance with the disclosed embodiments. Fig. 2 shows a flowchart of an example method in accordance with the disclosed embodiments. Fig. 3 shows an example of a general computer system according to the present disclosure. DETAILED DESCRIPTION

[0007] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or reduced to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for showing one skilled in the art how to variously employ the present invention.As those skilled in the art will appreciate, various features illustrated and described with reference to one of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The illustrated feature combinations represent representative embodiments for typical applications. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0008] Deep learning models have achieved remarkable results in various tasks, such as image classification, semantic segmentation, and time series prediction. However, deep learning models are also data-hungry, meaning they rely on a sufficiently large number of diverse samples to achieve good generalization. Unfortunately, large amounts of data (and diverse data) are not always available in many application areas, such as medical imaging applications. A popular solution to alleviate data scarcity is to use data augmentation to improve the size and quality of the training dataset.Numerous review and benchmark publications have investigated the effectiveness of data augmentation for various tasks and compared different data augmentations with regard to the resulting model quality through extensive empirical evaluations. These extensive empirical evaluations require a large amount of computational resources and intensive expert feedback. An efficient method for the automatic evaluation and assessment of data augmentation is still lacking.

[0009] Generative models are often trained to approximate the true distribution p*(x) of the training samples. The trained generative models provide an effective and efficient way to estimate the density p(x; θ) (with model parameters θ) of high-dimensional data, such as images.

[0010] One method for evaluating data augmentation is based on density estimation by generative models. It is expected that the model-estimated density can provide information about how close a sample is to the training data distribution p*(x). This means that the trained model predicts a high probability for samples that are close to the training data distribution and a lower probability for samples that are far from the training data distribution. However, this is not the case for density estimation in high-dimensional space. For example, a trained normalizing flow model predicts a higher probability for a completely black image (which was never observed during training) than for a training sample.

[0011] This is due to a problem of density estimation in high-dimensional space. A typical set of a generative model contains a large portion of the probability mass of a distribution, but not necessarily the highest density or probability points. This problem is often referred to as the Gaussian annulus theorem, which states that the samples are distributed around a sphere with radius d (typical set) in a d-dimensional isotropic Gaussian distribution. Thus, although the origin has the highest density, it is far from the typical set from which most samples are drawn.

[0012] The disclosed embodiments can evaluate data augmentations by measuring whether the augmented training samples are contained in or close to the typical set of a generative model trained on a training set. The typical set of a distribution is the set whose elements have an information content sufficiently close to that of the expected information of the distribution. The distance of a sample to the typical set can be specified with a typicality score. The typicality score can be estimated by the distance of the log density (log p(x; θ)) to the empirical entropy of a model as TS(x)=|−logp(x;θ)−H^p|, where TS(x) is the typing score of sample x, p(x; θ) is the density of sample x with respect to a model with parameters θ and Ĥ pis the estimated empirical entropy, which can be calculated over the training set as follows H^p=Ex∼Dtrain[−logp(x;θ)], where E is the expected value of the samples x from the training set D trainIf a sample has a low typing score, it is close to the empirical entropy of the model. That is, the lower the typing score, the closer the sample is to the empirical entropy of the model. Typing can be used to detect samples that fall outside the distribution. In some embodiments, samples that fall outside the distribution are considered unsuitable for inclusion in the training set. In some embodiments, typing can be used to evaluate data augmentation. In some disclosed embodiments, an augmented sample is considered suitable if its typing score is less than or equal to a threshold α. The threshold α can be estimated from the training samples (with an optional tunable parameter ε to control the acceptable range) as α=maxx∈DtrainTS(x)+ε where ε can have a default value of zero. By setting or adjusting ε, the user can adjust the tolerance range for the typing value based on their applications.

[0013] The presented embodiments use a typing score to evaluate data augmentations. Data augmentations suitable for training a machine learning model can be included in a training set useful for training the machine learning model. Data augmentations that are not suitable for training a machine learning model can be excluded from a training set useful for training the machine learning model.

[0014] Fig. Figure 1 shows a flowchart of an example method 100 in accordance with the disclosed embodiments. The method 100 can be used to determine whether an augmented training item is suitable for inclusion in a training set.

[0015] In operation 102, the method 100 estimates an empirical entropy of a training set on which a generative model has been trained. Examples of trained generative models that may be used in the method 100 are normalizing flows, diffusion models, and variational autocoders. The estimate of an empirical entropy may be calculated over the training set used to train the generative model. In disclosed embodiments, the estimate of the empirical energy Ĥ p as described above with respect to equation (2).

[0016] At operation 104, method 100 generates a typing score for the distance between 1) an augmented training item of a training item included in the training set and 2) a typing set of the trained generative model, wherein the typing score is based on the estimated empirical entropy. In disclosed embodiments, the typing score (also referred to as a typicality score) may be estimated as described above in accordance with equation (2).

[0017] In operation 106, the typing score may be compared to a threshold to determine whether the augmented training item is suitable for inclusion in the training set. In some disclosed embodiments, an augmented sample is classified as suitable for inclusion if its typing score is less than or equal to a threshold. In the disclosed embodiments, a threshold α may be estimated as described above according to equation (3).

[0018] In disclosed embodiments, each of a plurality of augmented training elements can be tested for suitability for inclusion in a training set. The plurality of augmented training elements can be generated using the same augmentation technique or different augmentation techniques. For example, each of the augmented training elements can be generated using the same augmentation technique, or each of the augmented training elements can be generated using a different augmentation technique. This means that the disclosed embodiments are not limited to specific augmentation techniques.

[0019] In disclosed embodiments, a plurality of suitable augmented training items may be included in an original training set to generate a second training set. The second training set may or may not contain all of the training items from the original training set. That is, one or more training items may be removed from the original training set to generate the second training set. In disclosed embodiments, the second training set may contain multiple identical augmented training items and / or multiple training items from the original set. This may be done, for example, when certain training items are to be emphasized when training a machine learning model on the second training set.

[0020] One or more machine learning models can be trained on the second training set. For example, a plurality of machine learning models can be trained on the second training set. The machine learning models to be trained can contain different machine learning algorithms. For example, the second training set can be used to train a machine learning model based on a neural network and also to train a machine learning model based on a support vector machine. This can be done, for example, to compare the effectiveness of different machine learning models.

[0021] Fig. Figure 2 shows a flowchart of an example method 200 in accordance with the disclosed embodiments. The method 200 can be used to generate a training set useful for training one or more machine learning models.

[0022] In operation 202, the method 200 estimates an empirical entropy of a training set on which a generative model has been trained. The estimated entropy can be generated in the same way as described above with respect to the Fig. 1 described operation 102.

[0023] In operation 204, the method 200 generates a typicality score for each of a plurality of augmented training samples. In operation 204, the typicality score is based on the estimated empirical entropy. The plurality of augmented training items may be generated by the same augmentation technique or by different augmentation techniques. The typicality scores may be generated in accordance with disclosed embodiments. For example, the typicality scores may be generated according to the method described in Fig. 1 shown operation 104.

[0024] In operation 206, the method 206 compares the typing scores generated for each of the plurality of augmented training items with a threshold to determine whether the augmented training item is suitable for inclusion in the training set. The suitability may be determined in accordance with disclosed embodiments. For example, the suitability may be determined according to the method described in Fig. 1 shown operation 106.

[0025] In operation 208, the method includes in the training set each of the plurality of augmented training items determined to be suitable for inclusion in the training set. In some disclosed embodiments, multiple identical augmented training items may be included in the training set. In some disclosed embodiments, one or more training items may be removed from the training set.

[0026] The training set containing the plurality of augmented training items can be used to train one or more machine learning models. For example, a plurality of machine learning models can be trained on the training set. The machine learning models to be trained can contain different machine learning algorithms. For example, the second training set can be used to train a machine learning model based on a neural network and also to train a machine learning model based on a support vector machine. This can be done, for example, to compare the effectiveness of different machine learning models.

[0027] Fig. 3 shows a block diagram of an exemplary embodiment of a general computer system 300. The computer system 300 may include a series of instructions that may be executed to cause the computer system 300 to perform one or more of the methods or computer-assisted functions disclosed herein. For example, the computer system 300 may include executable instructions to perform the Fig. 1 and Fig. 2. Computer system 300 may be connected to other computer systems or peripheral devices via a network. Furthermore, computer system 300 may include or be integrated with other computer devices.

[0028] As in Fig. 3, the computer system 300 may include one or more processors 302. The one or more processors 302 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. The computer system 300 may include main memory 304 and static memory 306 that may communicate with each other via a bus 308. As shown, the computer system 300 may further include a video display device 310, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display.Additionally, computer system 300 may include an input device 312, such as a remote control device with a wireless keypad, a keyboard, a microphone coupled to a speech recognition engine, a camera, such as a video camera or a still camera, or a cursor control device 314, such as a mouse. Computer system 300 may also include a hard disk drive unit 316, a signal generating device 318, such as a speaker, and a network interface device 320. Network interface 320 may enable computer system 300 to communicate with other systems via a network 328. For example, network interface 320 may enable machine learning system 120 to communicate with a database server (not shown) or a controller in a manufacturing system (not shown).

[0029] In some embodiments, as in Fig.3, the drive unit 316 may include one or more computer-readable media 322 in which one or more sets of instructions 324, e.g., software, may be embedded. The instructions 324 may, for example, embody one or more of the methods or functionality disclosed herein. In a particular embodiment, the instructions 324 may reside entirely or at least partially in main memory 304, static memory 306, and / or processor 302 during execution by the computer system 300. The main memory 304 and the processor 302 may also include computer-readable media.

[0030] In some embodiments, specialized hardware implementations, such as application-specific integrated circuits, programmable logic arrays, and other hardware devices, may be constructed to implement one or more of the methods or functions described herein. Applications that may encompass the devices and systems of the various embodiments broadly include a variety of electronic and computer systems. One or more embodiments described herein may implement functions using two or more specific, interconnected hardware modules or devices with associated control and data signals that may be communicated between and through the modules or as parts of an application-specific integrated circuit. Accordingly, the present system includes software, firmware, and hardware implementations, or combinations thereof.

[0031] Although the computer-readable medium is depicted as a single medium, the term "computer-readable medium" encompasses a single medium or multiple media, such as a centralized or distributed database and / or associated caches and servers, that store one or more sets of instructions. The term "computer-readable medium" also encompasses any medium capable of storing or encoding a set of instructions for execution by a processor, or that causes a computer system to perform one or more of the methods or functionalities disclosed herein.

[0032] In some embodiments, some or all of the computer-readable media are non-transitory media. In a particular, non-limiting, exemplary embodiment, the computer-readable medium may comprise solid-state storage, such as a memory card or other enclosure containing one or more non-volatile read-only memories. Further, the computer-readable medium may be random-access memory or other volatile, rewritable memory. In addition, the computer-readable medium may be a magneto-optical or optical medium, such as a disk or tape, or other storage device for capturing carrier wave signals, such as a signal conveyed over a transmission medium.

[0033] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The words used in the description are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As described above, the features of various embodiments may be combined to form further embodiments of the invention that may not be expressly described or illustrated.While various embodiments might be described as advantageous or preferred over other prior art embodiments or implementations with respect to one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system characteristics depending on the specific application and implementation. These characteristics may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as less desirable than other prior art embodiments or implementations with respect to one or more features are not outside the scope of the disclosure and may be desirable for certain applications.

Claims

[1] Method comprising: Estimating an empirical entropy of a training set on which a generative model was trained; Generating a typing score for the distance between 1) an augmented training item of a training item included in the training set and 2) a typing set of the trained generative model, wherein the typing score is based on the estimated empirical entropy; and Comparing the typing score with a threshold to determine whether the augmented training item is suitable for inclusion in the training set. [2] The method of claim 1, wherein the trained generative model is a normalizing flow, a diffusion model, or a variational autoencoder. [3] The method of claim 1, wherein the typing score is an estimate of the distance between the log density of the trained generative model, where x is the augmented training item. [4] The method of claim 1, wherein the typing score is an estimate according to TS(x)=|−logp(x;θ)−H^p|, where TS(x) is the typing score of sample x, p(x; θ) is the density of sample x with respect to the trained generative model with parameters θ and Ĥ p is an estimated empirical entropy of the trained generative model. [5] the method according to claim 4, wherein the estimated empirical energy Ĥ p is calculated over the training set as H^p=Ex∼Dtrain[−logp(x;θ)], where E x~Dtrain the expected value of the samples x from the training set D train . Is. [6] The method of claim 1, comprising determining that the augmented training sample is suitable for inclusion in the training set if the typing score of the augmented training sample is less than or equal to the threshold. [7] The method of claim 1, wherein the threshold is selected from the samples in the training set as α=maxx∈DtrainTS(x)+ε is estimated, where D train is the training set, x is a sample in the training set, and ε is an adjustable parameter. [8] The method of claim 1, further comprising: including the augmented training item in the training set if it is determined that the augmented training item is suitable for inclusion in the training set. [9] The method of claim 8, further comprising: training a machine learning model on the training set containing the augmented training item. [10] Non-transitory memory containing processor-executable instructions that, when executed by one or more processors, cause a system to perform operations to Estimating an empirical entropy of a training set on which a generative model was trained; Generating a typing score for the distance between 1) an augmented training item of a training item included in the training set and 2) a typing set of the trained generative model, wherein the typing score is based on the estimated empirical entropy; and Comparing the typing score with a threshold to determine whether the augmented training item is suitable for inclusion in the training set. [11] System comprising: one or more processors; and a non-transitory memory containing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations for Estimating an empirical entropy of a training set on which a generative model was trained; Generating a typing score based on the estimated empirical entropy for each of a plurality of augmented training items; Comparing the generated typing score with a threshold for each of the plurality of augmented training items to determine whether the augmented training item is suitable for inclusion in the training set; and Including each of the plurality of augmented training items determined to be suitable for inclusion in the training set into the training set. [12] The system of claim 11, wherein the estimated empirical energy Ĥ p is calculated over the training set as H^p=Ex∼Dtrain[−logp(x;θ)], where Ĥ p is the estimated empirical energy and where E x~Dtrain. the expected value of the samples x from the training set is D train . [13] The system of claim 12, wherein the typing score is an estimate according to TS(x)=|−logp(x;θ)−H^p|, where TS(x) is the typing score of sample x and p(x; θ) is the density of sample x with respect to the trained generative model with parameters θ. [14] The system of claim 11, wherein the threshold is estimated from the samples in the training set as α=maxx∈DtrainTS(x)+ε, where D train is the training set, x is a sample in the training set, and e is an adjustable parameter. [15] The system of claim 11, wherein the operations comprise: Training one or more machine learning models on the training set after the operation of ingesting augmented training items is completed. [16] The system of claim 15, wherein the one or more machine learning models comprise a plurality of different machine learning models.

Images