Data expansion evaluation and automatic training set improvement through typicality

Abstract

Data expansion evaluation and automatic training set improvement through typicality are provided. The disclosed embodiments include methods for evaluating extended training elements. Augmented training elements may be generated using different augmentation techniques. The disclosed embodiments may generate a training set useful for training a plurality of different machine learning models.

Description

Technical Field

[0001] The present disclosure relates generally to the field of artificial intelligence. More specifically, the disclosed embodiments relate to data augmentation methods useful for generating training sets for machine learning models. Background Art

[0002] Training and developing deep learning models often requires a large number of diverse and informative training samples in the training set that are useful for training deep learning models. However, in many applications, such as industrial inspection, automated optical inspection, and medical imaging applications, collecting high-quality samples can be expensive (e.g., in terms of time, manpower, and / or computing power). In practice, data-intensive deep learning methods can suffer from data scarcity. To alleviate data scarcity, data augmentation techniques are sometimes used and have been shown to be effective in a variety of tasks.

[0003] However, not all data augmentations are equally helpful. For example, a model trained on many strongly noisy images may make biased predictions about real-world scenes, because strongly noisy images may be different from real images (i.e., far from typical deep learning embeddings). Previously, a large body of work has conducted extensive empirical evaluations to find which data augmentation to use for which data domain on which task. This evaluation is usually done by comparing the performance of a specific model trained with various data augmentations. Summary of the invention

[0004] Repeating the training to evaluate each data augmentation type is time consuming and may waste computational resources. Furthermore, the evaluation results may be biased towards the task of interest. For example, an image classification task and a semantic segmentation task prefer different data augmentations. Therefore, there is a need in the industry for efficient methods to evaluate data augmentations that are suitable for inclusion in a training set useful for training one or more machine learning models.

[0005] The disclosed embodiments may differ from prior art solutions in at least two aspects. First, the disclosed embodiments may require only a single trained generative model to evaluate multiple data augmentations, in contrast to prior art approaches that require a different trained model to evaluate each different augmentation. Second, the disclosed embodiments may use statistics estimated by the generative model rather than the model performance for a specific task.

[0006] The disclosed embodiments include a method comprising: estimating an empirical entropy of a training set on which a generative model has been trained; generating a typicality score for a distance between 1) an augmented training element of the training elements included in the training set and 2) a typicality set of the trained generative model, wherein the typicality score is based on the estimated empirical entropy; and comparing the typicality score to a threshold value to determine whether the augmented training element is suitable for inclusion in the training set. Other disclosed embodiments include a non-transitory memory comprising processor-executable instructions that, when executed by one or more processors, cause a system to perform operations comprising: estimating an empirical entropy of a training set on which a generative model has been trained; generating a typicality score for a distance between 1) an augmented training element of the training elements included in the training set and 2) a typicality set of the trained generative model, wherein the typicality score is based on the estimated empirical entropy; and comparing the typicality score to a threshold value to determine whether the augmented training element is suitable for inclusion in the training set. Other disclosed embodiments include a system comprising: one or more processors; and a non-transitory memory comprising processor-executable instructions, which, when executed by the one or more processors, cause the system to perform operations comprising: estimating an empirical entropy for a training set on which a generative model has been trained; for each of a plurality of augmented training elements, generating a typicality score based on the estimated empirical entropy; for each of the plurality of augmented training elements, comparing the generated typicality score to a threshold to determine whether the augmented training element is suitable for inclusion in the training set; and including each of the plurality of augmented training elements determined to be suitable for inclusion in the training set in the training set. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A flow chart of an example method according to the disclosed embodiments is shown.

[0008] Figure 2 A flow chart of an example method according to the disclosed embodiments is shown.

[0009] Figure 3 An example embodiment of a general purpose computer system according to the present disclosure is shown. DETAILED DESCRIPTION

[0010] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to adopt the present invention in different ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any of the figures may be combined with the features shown in one or more other figures to produce embodiments that are not explicitly shown or described. The combination of features shown provides representative embodiments of typical applications. However, for specific applications or implementations, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0011] Deep learning models have achieved remarkable results in various tasks such as image classification, semantic segmentation, time series prediction, etc. However, deep learning models are also data-hungry, i.e., these models rely on a sufficiently large number of diverse samples to have good generalization performance. Unfortunately, big data (and diverse data) are not always available in many application areas, such as medical imaging applications. A popular solution to alleviate data scarcity is to use data augmentation to increase the size and quality of training datasets. Many surveys and benchmark papers have studied the effectiveness of data augmentation on various tasks and compared various data augmentations in terms of the performance of the models they produce through extensive empirical evaluations. Extensive empirical evaluations require a lot of computational resources and intensive expert feedback. Effective methods for automatically evaluating data augmentation are still lacking.

[0012] Generative models are usually trained to be close to the true distribution p of training samples. * (x). The trained generative model provides an effective and efficient way to estimate the density p(x; θ) (with model parameters θ) of high-dimensional data such as images.

[0013] One approach for evaluating data augmentation is based on density estimation via generative models. The density estimated by the model is expected to reveal how closely the sample is distributed relative to the training data distribution p * (x). That is, the trained model predicts high likelihood for samples close to the training data distribution and lower likelihood for samples far from the training data distribution. However, this is not the case for density estimation in high-dimensional space. For example, a trained normalized flow model predicts higher likelihood for a completely black image (never observed during training) than for the training samples.

[0014] This arises from the problem of density estimation in high-dimensional space. The typical set of generative models contains most of the probability mass of the distribution, but not necessarily the highest density or probability points. This problem is often referred to as the Gaussian Anulus theorem, which states that the sample concentration is in a d-dimensional isotropic Gaussian distribution with radius Therefore, although the origin has the highest density, it is far away from the typical set from which most samples are drawn.

[0015] The disclosed embodiments can evaluate data augmentation by measuring whether the augmented training sample is in or close to the typical set of the generative model trained on the training set. The typical set of a distribution is a set whose elements have information content that is sufficiently close to the expected information of the distribution. The distance of a sample to the typical set can be represented by a typicality score. The typicality score can be estimated by the distance of the logarithmic density (logp(x; θ)) and the empirical entropy of the model as:

[0016]

[0017] where TS(x) is the typicality score of sample x, p(x;θ) is the density of sample x with respect to the model with parameters θ, and is the estimated empirical entropy, which can be calculated from the training set as:

[0018]

[0019] Where E is from the training set D train The expected value of sample x obtained in . When a sample has a low typicality score, it is close to the empirical entropy of the model. That is, the lower the typicality score, the closer the sample is to the empirical entropy of the model. Typicality can be used to detect out-of-distribution samples. In some disclosed embodiments, out-of-distribution samples are considered unsuitable to be included in the training set. The disclosed embodiments may use typicality for data augmentation evaluation. In some disclosed embodiments, when the typicality score of the augmented sample is less than or equal to a threshold α, the augmented sample is considered suitable. The threshold α can be estimated from the training samples (using an optional adjustable parameter ∈ to control the acceptable region) as:

[0020]

[0021] where ∈ may have a default value of zero. By tuning or adjusting ∈, users may adjust the tolerance region to the typicality score based on their application.

[0022] The disclosed embodiments utilize typicality scores to evaluate data augmentation. Data augmentation determined to be suitable for training a machine learning model can be included in a training set useful for training machine learning. Data augmentation determined to be unsuitable for training a machine learning model can be excluded from a training set useful for training a machine learning model.

[0023] Figure 1 A flow chart of an example method 100 according to the disclosed embodiments is shown. The method 100 may be used to determine whether an augmented training element is suitable for inclusion in a training set.

[0024] At operation 102, method 100 estimates the empirical entropy of a training set on which a generative model has been trained. Examples of trained generative models that can be used in method 100 include normalized flow, diffusion models, and variational autoencoders. An estimate of the empirical entropy can be calculated using the training set used to train the generative model. In the disclosed embodiment, the empirical entropy An estimate of can be calculated as described above with respect to equation (2).

[0025] At operation 104, method 100 generates a typicality score for 1) the distance between the augmented training elements included in the training set and 2) the typicality set of the trained generative model, wherein the typicality score is based on the estimated empirical entropy. In the disclosed embodiment, the typicality score can be estimated as described above according to equation (2).

[0026] At operation 106, the typicality score can be compared to a threshold to determine whether the augmented training element is suitable for inclusion in the training set. In some disclosed embodiments, the augmented sample is determined to be suitable for inclusion when the typicality score of the augmented sample is less than or equal to the threshold. In disclosed embodiments, the threshold α can be estimated as described above according to equation (3).

[0027] In the disclosed embodiments, each of a plurality of augmented training elements may be evaluated for suitability for inclusion in a training set. The plurality of augmented training elements may be generated by the same augmentation technique or by different augmentation techniques. For example, each augmented training element may be generated by using the same augmentation technique, or each augmented training element may be generated by using a different augmentation technique. That is, the disclosed embodiments are not limited to a particular augmentation technique.

[0028] In the disclosed embodiments, multiple suitable augmented training elements may be included in the original training set to produce a second training set. The second training set may or may not include all of the training elements in the original training set. That is, one or more training elements in the original training set may be removed to produce the second training set. In the disclosed embodiments, the second training set may include multiple identical augmented training elements and / or multiple training elements from the original set. This may be done, for example, if it is intended to emphasize specific training elements when training a machine learning model on the second training set.

[0029] One or more machine learning models may be trained on the second training set. For example, multiple machine learning models may be trained on the second training set. The machine learning models to be trained may include different machine learning algorithms. For example, the second training set may be used to train a machine learning model based on a neural network, and may also be used to train a machine learning model based on a support vector machine. For example, this may be done to compare the efficacy of different machine learning models.

[0030] Figure 2 A flow chart of an example method 200 according to the disclosed embodiments is shown. The method 200 can be used to generate a training set useful for training one or more machine learning models.

[0031] At operation 202, method 200 estimates the empirical entropy of a training set on which a generative model has been trained. The estimated entropy may be calculated as described above with reference to Figure 1 Operation 102 is shown generated in the same manner as described.

[0032] At operation 204, method 200 generates a typicality score for each of the plurality of augmented training samples. At operation 204, the typicality score is based on the estimated empirical entropy. The plurality of augmented training elements may be generated by the same augmentation technique or by different augmentation techniques. The typicality score may be generated according to the disclosed embodiments. For example, the typicality score may be generated according to Figure 1 The operation 104 is shown to generate.

[0033] At operation 206, method 206 compares the typicality score generated for each of the plurality of augmented training elements to a threshold value to determine whether the augmented training element is suitable for inclusion in the training set. Suitability may be determined according to disclosed embodiments. For example, Figure 1 Operation 106 is shown to determine suitability.

[0034] In operation 208, the method includes each of the plurality of augmented training elements determined to be suitable for inclusion in the training set in the training set. In some disclosed embodiments, a plurality of identical augmented training elements may be included in the training set. In some disclosed embodiments, one or more training elements may be removed from the training set.

[0035] The training set including multiple augmented training elements can be used to train one or more machine learning models. For example, multiple machine learning models can be trained on the training set. The machine learning models to be trained can include different machine learning algorithms. For example, the second training set can be used to train a machine learning model based on a neural network, and can also be used to train a machine learning model based on a support vector machine. For example, this can be done to compare the efficacy of different machine learning models.

[0036] Figure 3 A block diagram of an example embodiment of a general purpose computer system 300 is shown. The computer system 300 may include a set of instructions that may be executed to cause the computer system 300 to perform any one or more of the methods or computer-based functions disclosed herein. For example, the computer system 300 may include an executable program to execute Figure 1 and 2 The computer system 300 may be connected to other computer systems or peripheral devices via a network. In addition, the computer system 300 may include other computing devices or be included in other computing devices.

[0037] like Figure 3 As shown, the computer system 300 may include one or more processors 302. The one or more processors 302 may include, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs), or both. The computer system 300 may include a main memory 304 and a static memory 306, which may communicate with each other via a bus 308. As shown, the computer system 300 may also include a video display unit 310, such as a liquid crystal display (LCD), a projection television display, a flat panel display, a plasma display, or a solid-state display. In addition, the computer system 300 may include an input device 312, such as a remote control device with a wireless keypad, a keyboard, a microphone coupled to a speech recognition engine, a camera such as a video camera or a still camera, or a cursor control device 314 such as a mouse device. The computer system 300 may also include a disk drive unit 316, a signal generating device 318 such as a speaker, and a network interface device 320. The network interface 320 may enable the computer system 300 to communicate with other systems via a network 328. For example, the network interface 320 may enable the machine learning system 120 to communicate with a database server (not shown) or a controller (not shown) in a manufacturing system.

[0038] In some embodiments, Figure 3 As depicted, disk drive unit 316 may include one or more computer-readable media 322 in which one or more sets of instructions 324, such as software, may be embedded. For example, instructions 324 may embody one or more methods or functions, such as those disclosed herein. In certain embodiments, instructions 324 may reside completely or at least partially within main memory 304, static memory 306, and / or processor 302 during execution by computer system 300. Main memory 304 and processor 302 may also include computer-readable media.

[0039] In some embodiments, dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays, and other hardware devices, can be constructed to implement one or more methods or functions described herein. Applications that may include the apparatus and systems of various embodiments may broadly include various electronic and computer systems. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices that utilize associated control and data signals that can communicate between or through modules, or as part of an application specific integrated circuit. Therefore, the present system encompasses software, firmware, and hardware implementations, or combinations thereof.

[0040] Although the computer-readable medium is shown as a single medium, the term "computer-readable medium" includes a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers storing one or more sets of instructions. The term "computer-readable medium" should also include any medium that can store or encode a set of instructions executed by a processor or cause a computer system to perform any one or more of the methods or functions disclosed herein.

[0041] In some embodiments, some or all of the computer-readable media will be non-transitory media. In a specific non-limiting exemplary embodiment, the computer-readable medium may include a solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. In addition, the computer-readable medium may be a random access memory or other volatile rewritable memory. In addition, the computer-readable medium may include a magneto-optical or optical medium, such as a disk or tape or other storage device, to capture a carrier signal, such as a signal transmitted through a transmission medium.

[0042] Although exemplary embodiments are described above, this does not mean that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive words, not restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As mentioned above, the features of the various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, strength, durability, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as being less desirable than other embodiments or prior art implementations in terms of one or more characteristics are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method comprising: estimating the empirical entropy of a training set on which the generative model has been trained; generating a typicality score for a distance between 1) an augmented training element of the training elements included in the training set and 2) a typicality set of the trained generative model, wherein the typicality score is based on the estimated empirical entropy; as well as The typicality score is compared to a threshold to determine whether the augmented training element is suitable for inclusion in the training set.

2. The method of claim 1, wherein the trained generative model is a normalizing flow, a diffusion model, or a variational autoencoder.

3. The method of claim 1, wherein the typicality score is an estimate of the distance between the log densities of the trained generative models, where x is an augmented training element.

4. The method of claim 1, wherein the typicality score is estimated according to the following formula: where TS(x) is the typicality score of sample x, p(x;θ) is the density of sample x with respect to the trained generative model with parameters θ, and is the estimated empirical entropy of the trained generative model.

5. The method according to claim 4, wherein the estimated empirical energy The calculation using the training set is as follows: in From the training set D train The expected value of the sample x obtained in . 6 . The method according to claim 1 , wherein when the typicality score of the augmented training sample is less than or equal to a threshold, the augmented training sample is determined to be suitable for inclusion in the training set.

7. The method according to claim 1, wherein the threshold is estimated based on samples in a training set as follows: Where D train is the training set, x is a sample in the training set, and ∈ is a tunable parameter.

8. The method according to claim 1, further comprising: When the augmented training element is determined to be suitable for inclusion in the training set, the augmented training element is included in the training set.

9. The method according to claim 8, further comprising: Train a machine learning model on a training set that includes augmented training elements.

10. A non-transitory memory comprising processor-executable instructions that, when executed by one or more processors, cause a system to perform operations comprising: estimating the empirical entropy of a training set on which the generative model has been trained; generating a typicality score for a distance between 1) an augmented training element of the training elements included in the training set and 2) a typicality set of the trained generative model, wherein the typicality score is based on the estimated empirical entropy; as well as The typicality score is compared to a threshold to determine whether the augmented training element is suitable for inclusion in the training set.

11. A system comprising: one or more processors; as well as a non-transitory memory including processor-executable instructions that, when executed by one or more processors, cause the system to perform operations comprising: estimating the empirical entropy of a training set on which the generative model has been trained; For each of the plurality of augmented training elements, generating a typicality score based on the estimated empirical entropy; For each of the plurality of augmented training elements, comparing the generated typicality score to a threshold value to determine whether the augmented training element is suitable for inclusion in the training set; and Each of the plurality of augmented training elements determined to be suitable for inclusion in the training set is included in the training set.

12. The system of claim 11, wherein the estimated empirical energy The calculation using the training set is as follows: in is the estimated empirical energy, and From the training set D train The expected value of the sample x obtained in .

13. The system of claim 12, wherein the typicality score is estimated according to the following formula: where TS(x) is the typicality score of sample x, and p(x;θ) is the density of sample x with respect to the trained generative model with parameters θ.

14. The system according to claim 11, wherein the threshold is estimated based on samples in a training set as follows: Where D train is the training set, x is a sample in the training set, and ∈ is a tunable parameter.

15. The system of claim 11, wherein the operations comprise: After operations including augmenting training elements are completed, one or more machine learning models are trained on the training set.

16. The system of claim 15, wherein the one or more machine learning models comprises a plurality of different machine learning models.

Images