Method for balancing unbalanced data for use in training a machine learning model

Abstract

Method for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set of categories as well as a second set of data items belonging to a second set of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items; the method comprising: for each data item in a source batch comprising a plurality of data items of unbalanced data, categorizing said data item into a category among a plurality of categories; storing each categorized data item in a respective priority queue of a plurality of priority queues, the respective priority queue corresponding with the category associated with said categorized data item; and sampling the plurality of priority queues without replacement in accordance with a target distribution in order to generate a target batch in statistical conformity with the target distribution.

Description

[0001] Method for balancing unbalanced data for use in training a machine learning model

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to training in machine learning. Particular embodiments relate to a method for balancing unbalanced data for use in training a machine learning model and optionally for controlling an enterprise resource planning, ERP, system, and corresponding computer programs, computer-readable storage media, and data processing apparatus.

[0004] BACKGROUND

[0005] It is a common problem in machine learning, in particular when training models using machine learning techniques, that data is unbalanced.

[0006] Data can be defined as ‘unbalanced’ if said data contains a first set of data items belonging to a first set of categories as well as a second set of data items belonging to a second set of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items.

[0007] In other words, data can be defined as ‘unbalanced’ if said data comprises numerous data items belonging to just a few categories (i.e. a narrow peak) while comprising few data items belonging to many more categories (i.e. a long tail).

[0008] This problem especially, but not only, occurs in enterprise resource planning, ERP, systems tasked with managing a plurality of stock keeping units, SKUs, where some SKUs may be very popular (i.e. the narrow peak) while many other SKUs are only sold in very low quantities (i.e. the long tail). In US2020 / 202171A1 a system and a method for rapidly building, managing and sharing machine learning models are provided. This disclosure pertains to generating production annotations in unlabelled dataset.

[0009] SUMMARY

[0010] It is an aim of at least some embodiments according to the present disclosure to address this shortcoming of unbalanced data to any given target distribution.

[0011] Accordingly, there is provided in a first aspect of the present disclosure a computer- implemented method for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set of categories as well as a second set of data items belonging to a second set of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items; the method comprising:

[0012] - for each data item in a source batch comprising a plurality of data items of unbalanced data, categorizing said data item into a category among a plurality of categories;

[0013] - storing each categorized data item in a respective priority queue of a plurality of priority queues, the respective priority queue corresponding with the category associated with said categorized data item; and

[0014] - sampling the plurality of priority queues without replacement in accordance with a target distribution in order to generate a target batch in statistical conformity with the target distribution.

[0015] This method allows for any dataset to be sampled to a given target distribution, such that the desired and adequate sampling, training and further use of the balanced data can be performed.

[0016] In various embodiments, the method comprises, in order to construct the source batch: - obtaining from the unbalanced data a next data item N times until a new source batch containing M different data items is created, wherein N is equal to or greater than M.

[0017] This may allow to decouple the input of unbalanced data (which can be regarded as a so-called publish procedure) from the actual training of the machine learning model (which can use a so-called subscribe procedure). This may help to improve randomness, which may be beneficial for the training performance. Moreover, this decoupling may allow to scale both procedures more independently from each other.

[0018] Of course, alternatively, N may also be strictly greater than M.

[0019] In various further developed embodiments, the unbalanced data is provided for said obtaining as a stream of tensors; said obtaining comprises slicing an index from a tensor of the stream; and said storing of each categorized data item comprises storing the index of said categorized data item.

[0020] In various embodiments, the number of the plurality of priority queues is at least equal to the number of the plurality of categories, such that each category is associated with at least one priority queue.

[0021] In various embodiments, the priority of a data item in a priority queue of the plurality of priority queues is based at least on a freshness or staleness of said data item relating to how many times said data item has been sampled previously to generate target batches.

[0022] In this context, the staleness of a data item may indicate how many times said data item has been sampled previously to generate target batches, and the freshness of a data item may indicate the inverse or opposite. In other words, the freshness of the data item may decrease when the data item is sampled more, while the staleness for the data item may increase when the data item is sampled more.

[0023] In further developed embodiments, the priority of a data item in a priority queue of the plurality of priority queues may also or instead be based at least on a measure of how likely it is to contribute downstream to a good prediction, which can be estimated via a suitable objective function configured to take into account one or more statistical rules (and optionally one or more business rules). Such predicting mechanism may also be known in the art as active-learning.

[0024] The staleness and / or freshness of a data item may for instance, in another example, by extended with additional feedback (active-learning) for the model in terms of uncertainty-estimates. This may be implemented as a feedback signal that informs that the queues have not seen enough data of a certain subslice, which feedback information may be given as a score in combination with the already existing freshness-priority-score. It should be noted though, that the at least one measure of how likely to contribute to a good prediction may (next to uncertainty) be diversity, representativeness, or informativeness of the data.

[0025] In various embodiments, the method comprises discarding one or more stored data items with a lowest priority from a particular priority queue of the plurality of priority queues if that particular priority queue would otherwise overflow due to adding one or more new data items to that particular priority queue.

[0026] In various embodiments, the step of sampling the plurality of priority queues comprises sampling data items from the plurality of priority queues in such a way that there is a higher probability of sampling a data item from a priority queue associated with the first set of categories than from a priority queue associated with the second set of categories.

[0027] This may amount to undersampling of the narrow peak and oversampling of the long tail, as the first set of categories is also termed the long tail and the second set of categories is also termed the narrow peak, so giving higher probability to the first set prioritizes sampling from the long tail with respect to sampling from the narrow peak. The strategies of undersampling and oversampling and the well-balanced combination thereof allow the method according to the disclosure to efficiently match the desired target distribution. In various embodiments, the step of sampling the plurality of priority queues is performed such that oversampling within a same data batch is circumvented. This may ensure the uniqueness of the intra-batch data and prevent adding a bias into the method.

[0028] In various embodiments, the machine learning model is trained using a gradient descent optimization.

[0029] In various embodiments, the method is performed repeatedly, and comprises reusing the same plurality of priority queues over multiple iterations of the method.

[0030] Advantageously, this allows even the priority queues associated with the first set of categories (i.e. the priority queues of categories of the long tail) to fill up, even though there is less data in each category of the long tail than there is in the narrow peak).

[0031] In various further developed embodiments, the method may comprise augmenting, preferably just-in-time (which means that the augmenting is postponed until it is necessary, as opposed to a more eager approach), any data item to be sampled into the target batch if said data item has already been sampled in a previous iteration of the method.

[0032] In this manner, the salience of data items for downstream training may be improved, in order to improve the quality of downstream trained ML models.

[0033] Of course, the augmenting may be subject to and / or in accordance with one or more statistical rules and / or one or more business rules.

[0034] Furthermore, augmenting any data item, especially in the case of oversampling, allows the method to account for unintentionally added bias. A randomized augmentation function for a datatype may be applied over any data-item passing through oversampling (in practise these are the minority groups), before being placed back on the priority queue with updated priority. This ensures that the training algorithm never sees identical data-items even though it is oversampling the data poule. In various embodiments, each priority queue of the plurality of priority queues may comprise a heap.

[0035] In various embodiments, the method is moreover suitable for controlling an enterprise resource planning, ERP, system tasked with managing a plurality of stock keeping units, SKUs, of at least one stock inventory; the method comprising:

[0036] - obtaining a plurality of historical data elements, wherein each historical data element is associated with a historical performance value of an SKU of the plurality of SKUs;

[0037] - determining a plurality of predicted data elements for the plurality of SKUs, based on the plurality of historical data elements and using the generated target batch, wherein each predicted data element is associated with a predicted future performance value of an SKU of the plurality of SKUs; and

[0038] - providing at least one recommendation pertaining to the plurality of SKUs to the ERP system, based on the plurality of predicted data elements, in order to trigger the ERP system to affect the recommendation.

[0039] In this context, the at least one stock inventory may refer to physical (“brick and mortar”) shopping stores as well as online retail channels.

[0040] Thus, the method ensures that the ERP system is directly updated, which reduces interference in other processes, and increases responsiveness.

[0041] ERP systems are typically subjected to two taxonomies being:

[0042] 1. The product-taxonomy spanning from the entire catalogue to specific products at a specific size (SKU)

[0043] 2. The location-taxonomy spanning from retailer-wide-international to a specific physical store (location).

[0044] It is known in the field that making prediction of individual SKU for an individual location is very hard, because the quantities that are sold can be very low (known as sparse or zero-inflated), have high noise / variance and because they are discrete, and because of a high imbalance in the target variable. The above-described method is particularly suitable for making recommendations on the location-SKU or location-product level. The method allows for achieving greater accuracy by making forecasts on the lower level, instead of forecasting on higher aggregations in the taxonomy and extrapolating the finding to the low-level.

[0045] Additionally, there is provided in another aspect of the present disclosure a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding claim.

[0046] Additionally, there is provided in another aspect of the present disclosure a computer- readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1-9.

[0047] Additionally, there is provided in another aspect of the present disclosure a data processing apparatus for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set of categories as well as a second set of data items belonging to a second set of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items; the data processing apparatus comprising means for carrying out the method of any one of the above-described embodiments of the method for balancing unbalanced data for use in training a machine learning model.

[0048] In various embodiments, the data processing apparatus is moreover suitable for controlling an enterprise resource planning, ERP, system; the data processing apparatus comprising means for carrying out the above-described method of controlling an ERP system.

[0049] The embodiments described herein are provided for illustrative purposes and should not be construed as limiting the scope of the invention. It is to be understood that the invention encompasses other embodiments and variations that are within the scope of the appended claims. The invention is not restricted to the specific configurations, arrangements, and features described herein. The invention has wide applicability and should not be limited to the specific examples provided. The embodiments disclosed are merely exemplary, and the skilled person will appreciate that various modifications and alternative designs can be made without departing from the scope of the invention.

[0050] BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In the following description, a number of exemplary embodiments will be described in more detail, to help understanding, with reference to the appended drawings, in which: Figure 1 schematically illustrates a first embodiment of the method according to the present disclosure;

[0052] Figure 2 schematically illustrates a second embodiment of the method according to the present disclosure; and

[0053] Figure 3 schematically illustrates a third embodiment of the method according to the present disclosure.

[0054] DETAILED DESCRIPTION

[0055] Figure 1 schematically illustrates a first embodiment of the method according to the present disclosure.

[0056] The figure shows the computer-implemented method for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set 14 of categories as well as a second set of data items belonging to a second set 13 of categories distinct from the first set of categories, wherein the first set 14 of categories is more numerous than the second set 13 of categories and the second set of data items is more numerous than the first set of data items.

[0057] The figure shows an example of such an unbalanced source batch 10 of data comprising a plurality of data items of unbalanced data, including a narrow peak of numerous data items belonging to just a few categories 13 on the left side of the graph (plotting categories to amounts), while comprising a long tail of few data items belonging to many more categories 14 on the right side of the graph. For the sake of clarity, several categories of data items in the narrow peak have been indicated with “to be undersampled”, in order to indicate that later on in the method they will be undersampled. Likewise, several categories of data items in the long tail have been indicated with “to be oversampled”, in order to indicate that later on in the method they will be oversampled.

[0058] The method may comprise:

[0059] - for each data item in a source batch 10 comprising a plurality of data items of unbalanced data, categorizing said data item into a category among a plurality of categories.

[0060] Optionally, this step may have been performed beforehand, and the relevant category for each data item may be stored in some memory.

[0061] In more detail, the categorizing may optionally use a categorization function configured for categorizing a data item to a respective bin of a plurality of bins, the respective bin being associated with said data item.

[0062] The method may comprise:

[0063] - storing each categorized data item in a respective priority queue of a plurality of priority queues 15, the respective priority queue corresponding with the category associated with said categorized data item.

[0064] The method may comprise:

[0065] - sampling the plurality of priority queues without replacement in accordance with a target distribution 16 in order to generate the target batch 17 in statistical conformity with the target distribution.

[0066] Preferably, the step of sampling the plurality of priority queues comprises sampling data items from the plurality of priority queues in such a way that there is a higher probability of sampling a data item from a priority queue associated with the first set 14 of categories than from a priority queue associated with the second set 13 of categories. As indicated above, several categories of data items in the narrow peak 13 have been undersampled, in the sense that they are less likely to be present in the target batch 17 than they were in the source batch 10. Likewise, several categories of data items in the long tail 14 have been oversampled in the sense that they are more likely to be present in the target batch 17 than they were in the source batch 10.

[0067] The target distribution 16 is shown in another graph in the figure, and it can be seen that the target distribution 16 has a less steep exponential downward curve, meaning that relatively more weight can be attributed to data items in the tail section 14, at the trade-off cost of attributing relatively less weight to data items in the head section 13 (i.e. the section corresponding with the narrow peak).

[0068] The figure shows that there are multiple priority queues, in order to allow the step of sampling to conform statistically (to some degree of approximation) with the target distribution. Preferably, there may be exactly one priority queue per category.

[0069] The figure further shows a functional element, called an “Under / Over Sampler”, or “UOSampler” in short, configured for said sampling to generate the target batch 17.

[0070] It can be seen that the generated target batch 17 resembles the general statistics of the target distribution 16 more closely, in the sense that it does not feature the steep exponential downward curve of the unbalanced source batch 10. In this sense, the unbalanced data has been balanced (or at least more balanced than the source batch).

[0071] Figure 2 schematically illustrates a second embodiment of the method according to the present disclosure.

[0072] The figure shows an optional additional development to the first embodiment of Figure 1. In Figure 2, the method is shown to optionally comprise, in order to construct the source batch 10 of Figure 1 (here source batch 20): obtaining from the unbalanced data a next data item N times until a new source batch 21 (also called a ‘Minibatch’) containing M different data items is created, wherein N is equal to or greater than M. As in Figure 1 , the method illustrated here results in a balanced (or at least more balanced) target batch 23.

[0073] Preferably, afterwards, the freshness or staleness of data items (or ‘data points’) that have been sampled may be updated, preferably decreased, e.g. decremented by 1 , or increased, e.g. incremented by 1 , respectively, in order to show that those data items have already been used in an iteration of the method. This is especially useful if the method is repeated over multiple iterations.

[0074] Preferably, after updating the freshness or staleness of the sampled data items, the priority queues 22 may be refilled with the sampled data items. Preferably, this is done in a manner that allows discarding one or more stored data items with a lowest priority from a particular priority queue of the plurality of priority queues 22 if that particular priority queue would otherwise overflow due to adding one or more new data items to that particular priority queue.

[0075] The above two steps (of updating freshness / staleness and of refilling with preferable discarding) are referred to with the reference number 24, which may thus be summarized as ‘reusing’ the priority queues.

[0076] Preferably, the method may also comprise augmenting (i.e. synthetically adapting), preferably just-in-time, any data item to be sampled into the target batch if said data item has already been sampled in a previous iteration of the method.

[0077] In various embodiments, the number of the plurality of priority queues 22 is at least equal to the number of the plurality of categories, such that each category is associated with at least one priority queue.

[0078] Preferably, there may be exactly one priority queue per category.

[0079] In various embodiments, the priority of a data item in a priority queue of the plurality of priority queues 22 is based at least on a freshness or staleness of said data item relating to how many times said data item has been sampled previously to generate the current target batch and / or previous target batches.

[0080] In this context, the freshness of a data item may be taken to relate to how many times said data item has been sampled previously to generate previous target batches, in the sense that the more times a particular data item has been sampled, the lower its freshness will be. In a further developed embodiment, this technical relationship can be represented using a related parameter, namely a staleness, which can for example be initialized at zero and can be incremented each time the particular data item is sampled to generate target batches. In this manner, if a data item has already contributed to many target batches, it becomes less likely for it to be included in new target batches.

[0081] In various embodiments, the method comprises discarding one or more stored data items with a lowest priority from a particular priority queue of the plurality of priority queues if that particular priority queue would otherwise overflow due to adding one or more new data items to the priority queue.

[0082] In various embodiments, the step of sampling the plurality of priority queues comprises sampling data items from the plurality of priority queues 22 in such a way that there is a higher probability of sampling a data item from a priority queue associated with the first set 14 of categories than from a priority queue associated with the second set 13 of categories.

[0083] In various embodiments, the machine learning model is trained using a gradient descent optimization.

[0084] In various embodiments, the method is performed repeatedly, and comprises reusing the same plurality of priority queues over multiple iterations of the method.

[0085] Figure 3 schematically illustrates a third embodiment of the method according to the present disclosure. The figure shows an exemplary embodiment of the overall method, termed here the Data Pipeline Process (because the method can serve to supply data to a downstream process, and thus serve as a data pipeline, see below), and indicates that this method can optionally be repeated, optionally on many machines in order to scale up processing.

[0086] The exemplary method is shown to comprise reading from a source stream into the so-termed Under Over Sampler, which refers to the above-described element of undersampling from the narrow peak categories and oversampling from the long tail categories.

[0087] The figure shows that the Under Over Sampler may be interfacing input and output with a per-category (preferably pre-allocated) memory tensor, via read and write operations of the actual data items and categories (or, equivalently, category bins) on the one hand, and with a per-category priority queue heap, via read and write operations of the indexes of memory, the priority scores (for the priority queues), and the category bins (or, equivalently, the categories), on the other hand. The relationship between the priority queues and the memory tensors (both of which preferably number as many as there are categories), is that the priority queues may preferably store the indexes of the memory tensors.

[0088] The figure further shows that the data pipeline process can be effectively decoupled from a downstream consumer process, in this case a decoupled machine learning training process, by respectively writing to or reading from a shared data buffer, which may randomly be read from (preferably removing data that is read) by the decoupled training process.

[0089] As used in this application and in the claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly dictates otherwise. The systems, apparatus, and methods described herein should not be construed as limiting in any way. Instead, the present disclosure is directed toward all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub-combinations with one another. The disclosed systems, methods, and apparatus are not limited to any specific aspect or feature or combinations thereof, nor do the disclosed systems, methods, and apparatus require that any one or more specific advantages be present or problems be solved. Any theories of operation are to facilitate explanation, but the disclosed systems, methods, and apparatus are not limited to such theories of operation.

[0090] Although the operations of some of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this manner of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed systems, methods, and apparatus can be used in conjunction with other systems, methods, and apparatus. Additionally, the description sometimes uses terms like “obtaining” and “outputting” to describe the disclosed methods. These terms are high-level abstractions of the actual operations that are performed. The actual operations that correspond to these terms will vary depending on the particular implementation and are readily discernible by the skilled person.

[0091] It will be appreciated that for simplicity and clarity of illustration, where appropriate, reference numerals may have been repeated among the different figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by the skilled person that the examples described herein can be practiced without these specific details. In other instances, methods, procedures and components have not been described in detail so as not to obscure the related relevant feature being described. The drawings are not necessarily to scale and the proportions of certain parts may be exaggerated to better illustrate details and features. The description is not to be considered as limiting the scope of the examples described herein.

[0092] Of course, the skilled person will understand that the present invention may be implemented in other ways than those specifically set forth herein without departing from the essential characteristics of the invention. The embodiments described herein are thus to be considered in all respects as illustrative and not restrictive, and all changes within the scope of the appended claims are intended to be embraced therein.

Claims

CLAIMS1. A computer-implemented method for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set (14) of categories as well as a second set of data items belonging to a second set (13) of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items; the method comprising:- for each data item in a source batch (10, 20) comprising a plurality of data items of unbalanced data, categorizing said data item into a category among a plurality of categories;- storing each categorized data item in a respective priority queue of a plurality of priority queues (15, 22), the respective priority queue corresponding with the category associated with said categorized data item; and- sampling the plurality of priority queues without replacement in accordance with a target distribution (16) in order to generate a target batch (17, 23) in statistical conformity with the target distribution.

2. The method of claim 1 , comprising, in order to construct the source batch:- obtaining from the unbalanced data a next data item N times until a new source batch (21) containing M different data items is created, wherein N is equal to or greater than M.

3. The method of claim 2, wherein the unbalanced data is provided for said obtaining as a stream of tensors; wherein said obtaining comprises slicing an index from a tensor of the stream; and wherein said storing of each categorized data item comprises storing the index of said categorized data item.

4. The method of any preceding claim, wherein the number of the plurality of priority queues is at least equal to the number of the plurality of categories, such that each category is associated with at least one priority queue.

5. The method of any preceding claim, wherein the priority of a data item in a priority queue of the plurality of priority queues is based at least on a freshness or staleness of said data item relating to how many times said data item has been sampled previously to generate target batches.

6. The method of any preceding claim, comprising discarding one or more stored data items with a lowest priority from a particular priority queue of the plurality of priority queues if that particular priority queue would otherwise overflow due to adding one or more new data items to that particular priority queue.

7. The method of any preceding claim, wherein the step of sampling the plurality of priority queues comprises sampling data items from the plurality of priority queues in such a way that there is a higher probability of sampling a data item from a priority queue associated with the first set (14) of categories than from a priority queue associated with the second set (13) of categories.

8. The method of any preceding claim, wherein the machine learning model is trained using a gradient descent optimization.

9. The method of any preceding claim, performed repeatedly, and comprising reusing (24) the same plurality of priority queues over multiple iterations of the method.

10. The method of claim 9, comprising augmenting, preferably just-in-time, any data item to be sampled into the target batch if said data item has already been sampled in a previous iteration of the method.

11. The method of any preceding claim, for controlling an enterprise resource planning, ERP, system tasked with managing a plurality of stock keeping units, SKUs, of at least one stock inventory; the method comprising:- obtaining a plurality of historical data elements, wherein each historical data element is associated with a historical performance value of an SKU of the plurality of SKUs;- determining a plurality of predicted data elements for the plurality of SKUs, based on the plurality of historical data elements and using the generated target batch, whereineach predicted data element is associated with a predicted future performance value of an SKU of the plurality of SKUs; and- providing at least one recommendation pertaining to the plurality of SKUs to the ERP system, based on the plurality of predicted data elements, in order to trigger the ERP system to effect the recommendation.

12. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any preceding claim.

13. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1-11.

14. A data processing apparatus for balancing unbalanced data for use in training a machine learning model, wherein data is defined to be unbalanced if said data contains a first set of data items belonging to a first set of categories as well as a second set of data items belonging to a second set of categories distinct from the first set of categories, wherein the first set of categories is more numerous than the second set of categories and the second set of data items is more numerous than the first set of data items; the data processing apparatus comprising means for carrying out the method of any one of claims 1-10.

15. The data processing apparatus of claim 14, for controlling an enterprise resource planning, ERP, system; the data processing apparatus comprising means for carrying out the method of claim 11.

Images