Reducing class imbalance in machine learning training datasets

Abstract

Class imbalance in a training dataset can adversely affect the accuracy of a machine learning model in classifying rare events that are under-represented in the training dataset. Training datasets that include time series data present unique challenges. Thus, resampling techniques are disclosed for upsampling and / or downsampling a training dataset of time series. Upsampling can take into account the temporal correlation of time samples in the time series while generating synthetic time series that mimic the feature values ​​of the time series that belong to the minority class. Downsampling can be used to fine-tune the ratio of time series that belong to the minority class to the majority class.

Description

[Technical field]

[0001] background Field of Disclosure FIELD OF THE DISCLOSURE

[0002] Embodiments described herein relate generally to machine learning, and more particularly to reducing class imbalance in datasets used to train machine learning models. [Background technology]

[0002] 2. Description of Related Art Many mission-critical applications require the successful identification of rare events so that appropriate action can be taken. Power system protection is one such application. In particular, when a fault occurs in the power system, protective devices may be utilized to disconnect the faulted portion of the power system from the rest of the power system in order to contain the fault. Alternatively, monitoring devices may be utilized to identify anomalous behavior within the power system. It should be appreciated that other types of mission-critical applications have similarly rare events for which response actions must be selected and taken.

[0003] Machine learning is one useful tool for responding to events in systems, including mission-critical applications. Machine learning models are trained to generate outputs (e.g., classifications) based on input features. Because machine learning is data-driven, the quality of the dataset used to train the model is a key factor in the accuracy of the resulting model. In supervised learning of machine learning classifiers, the training dataset includes a set of feature values ​​(e.g., feature vectors) labeled by their corresponding ground truth classes.

[0004] When the possible classes include a class that occurs relatively rarely, the distribution of labels in the training dataset will be heavily skewed away from the rare class and towards other classes. In other words, the rare class may be significantly underrepresented in the training dataset, resulting in an imbalanced training dataset. This imbalance may adversely affect the performance of any machine learning model trained on the dataset. Summary of the Invention [Means for solving the problem]

[0005] overview Thus, a system, method, and non-transitory computer-readable medium for reducing class imbalance in a training dataset for machine learning are disclosed. An objective of the embodiments is to increase the proportion of minority-class time series in the training dataset by generating a minority-class synthetic time series and / or reducing the number of majority-class time series, thereby reducing class imbalance in the training dataset (e.g., within an acceptable range). A further objective of some embodiments is to generate minority-class synthetic time series using adjacent time series for each sampling of time series in the training dataset, for example, based on the distance between time samples in a pair of time series. A further objective of some embodiments is to train a machine learning algorithm, such as a binary classifier, using a training dataset in which class imbalance has been reduced by the disclosed technique. Advantageously, the reduction of class imbalance in the training dataset results in higher accuracy in machine learning classifiers trained with a more balanced training dataset.

[0006] In one embodiment, a method is disclosed for reducing class imbalance in a training dataset for machine learning, the method including using at least one hardware processor to execute a process for receiving at least a subset of the training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value for each of one or more features (e.g., a plurality of features) at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; for each time series in the at least the subset of the training dataset, identifying a number of neighboring time series in the at least the subset of the training dataset based on a distance metric; for each of one or more of the neighboring time series, for each of the one or more features and for each of the plurality of labeled time samples in the neighboring time series, generating a synthetic value of the feature in a synthetic time sample; generating a synthetic time series from the synthetic time samples including the synthetic values ​​of the one or more features; and adding the generated synthetic time series to the training dataset.

[0007] In a further embodiment, a method for detecting an event in an electrical system includes using at least one hardware processor to execute a process for reducing class imbalance of a training dataset for a machine learning algorithm, the process comprising: receiving at least a subset of the training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; and, for each time series in the at least the subset of the training dataset, calculating a distance metric based on a distance metric for the at least the subset of the training dataset. a plurality of adjacent time series in a data set; for each of one or more of the adjacent time series, for each of one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including the composite values ​​of the one or more features; adding the generated synthetic time series to a training dataset; training a machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset; executing the trained machine learning algorithm on acquired data related to the electrical system; and detecting an event based on the execution of the trained machine learning algorithm.

[0008] The event may be an abnormality in the electrical system, in particular a failure of at least one transmission line of the electrical system.

[0009] The acquired data may have the same structure as the data training data (set). The acquired data may have the same features and / or labels as the training data (set). In other words, the acquired data may (also) include a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including, for each time series in at least a subset of the training data set, a value of each of one or more features at a particular time point in the time frame of the time series, and a label representing one of a plurality of classes into which the time sample was classified. However, it will be understood by those skilled in the art that these time series are not identical to the time series of the training data set. In particular, the proportion of representative classes of the data training data (set) and representative classes of the acquired data may be different. For example, the observed data set may be unbalanced while the training data set is balanced by the above method. The acquired data may include a plurality of time series or one or more time series. The acquired data may relate to the same electrical system or the same electrical system category as the training data.

[0010] The method may further include, after adding the generated synthetic time series to the training dataset and before training the machine learning algorithm, removing one or more majority time series from the training dataset, wherein a majority of the plurality of labeled time samples include labels that do not represent a minority class.

[0011] The elimination may be performed until a maximum number of iterations is reached. The pruning may be performed until the ratio between the number of minority time series in the training dataset and the number of majority time series in the training dataset is within a threshold tolerance, where a minority time series is defined as a time series in which a majority of the labeled time samples include a label representing a minority class.

[0012] Receiving at least a subset of the training dataset may include selecting a random sample of a predetermined number of time series from the training dataset.

[0013] Receiving at least a subset of the training dataset may include selecting a sample of a predetermined number of time series from the training dataset such that the training dataset includes a higher proportion of minority time series than non-minority time series, where a minority time series is defined as a time series in which a majority of the plurality of labeled time samples include a label that represents a minority class, and a non-minority time series is defined as a time series in which a majority of the plurality of labeled time samples include a label that represents one of a plurality of classes that is not the minority class.

[0014] The multiple classes may consist of two classes. The number of adjacent time series may be a predetermined number greater than one.

[0015] The method may further include using at least one hardware processor to generate a predetermined number of synthetic time series for each time series in at least the subset of the training data set. Generating the predetermined number of synthetic time series may include randomly selecting one of the identified number of adjacent time series for use in generating the synthetic time series until the predetermined number of synthetic time series are generated.

[0016] For each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature may include generating a composite value of the feature according to a first bias toward the minority class when a label of a corresponding labeled time sample in the time series represents a minority class of the plurality of classes, and generating a composite value of the feature according to a second bias toward the minority class less than the first bias when a label of a corresponding labeled time sample in the time series does not represent the minority class. The minority class may be defined as one class of the plurality of classes that is less likely to be represented by a label in the plurality of time samples in the plurality of time series in the training dataset than at least one other class of the plurality of classes. Generating a composite value of the feature according to the first bias may include adding a randomly generated value to a value of the feature in the corresponding labeled time sample in the time series. Generating a composite value of the feature according to the second bias may include calculating a distance between a value of the feature in a corresponding labeled time sample in the time series and a value of the feature in a labeled time sample in an adjacent time series; randomly generating a scaling factor; calculating a scaling value by multiplying the calculated distance by the randomly generated scaling factor; and adding the scaling value to the value of the feature in the corresponding labeled time sample in the time series.

[0017] For each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature may include generating a composite value of the feature based on a value of the feature in the corresponding labeled time sample in the time series when the label of the corresponding labeled time sample in the time series represents a minority class of the plurality of classes, and generating a composite value of the feature based on both a value of the feature in the corresponding labeled time sample in the time series and a value of the feature in the labeled time sample in the adjacent time series when the label of the corresponding labeled time sample in the time series does not represent the minority class. The minority class may be defined as one class of the plurality of classes that is less likely to be represented by a label in the plurality of time samples in the plurality of time series in the training dataset than at least one other class of the plurality of classes. For example, the minority class may be defined as one class of the plurality of classes that is least likely to be represented by a label in the plurality of time samples in the plurality of time series in the training dataset. Generating a composite value of the feature based on a value of the feature in the corresponding labeled time sample in the time series may include adding a randomly generated value to a value of the feature in the corresponding labeled time sample in the time series. Generating a composite value of the feature based on both a value of the feature in a corresponding labeled time sample in the time series and a value of the feature in a labeled time sample in an adjacent time series may include calculating a distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series, randomly generating a scaling factor, calculating a scaling value by multiplying the calculated distance by the randomly generated scaling factor, and adding the scaling value to the value of the feature in the corresponding labeled time sample in the time series.

[0018] When the labels of corresponding labeled time samples in a time series do not represent a minority class and the labels of labeled time samples in an adjacent time series do represent a minority class, the range of scaling values ​​may be constrained to produce scaling values ​​that are closer to the values ​​of the feature in the labeled time samples in the adjacent time series than the values ​​of the feature in the corresponding labeled time samples in the time series.When the labels of corresponding labeled time samples in a time series do not represent a minority class and the labels of labeled time samples in an adjacent time series do not represent a minority class, the range of scaling values ​​may be constrained to produce scaling values ​​that are closer to the values ​​of the feature in the corresponding labeled time samples in the time series than the values ​​of the feature in the corresponding labeled time samples in the adjacent time series.

[0019] Generating the synthetic time series may include labeling the synthetic time series with a label value derived based on a label representing one of a plurality of classes into which the time series is classified and a label representing one of a plurality of classes into which the adjacent time series is classified. The label value may be further derived based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and the adjacent time series. When the plurality of classes consists of two classes, the label value may be derived by binarizing a value based on the distance between the synthetic time series and the time series and the distance between the synthetic time series and the adjacent time series.

[0020] The method may further include using at least one hardware processor to train a machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset.

[0021] The one or more features may represent parameters of the electrical system, and at least one of the plurality of classes represents an action to be taken in response to an anomaly in the electrical system.

[0022] The method may further include using at least one hardware processor to randomly select, for each time series in at least the subset of the training dataset, one or more neighboring time series from the identified number of neighboring time series.

[0023] The method may further include using at least one hardware processor to remove from the training dataset one or more majority time series where a majority of the plurality of labeled time samples include labels that do not represent the minority class, until a ratio between a number of minority time series in the training dataset where a majority of the plurality of labeled time samples include labels that represent the minority class and a number of majority time series in the training dataset where a majority of the plurality of labeled time samples include labels that represent the majority class, over one or more iterations, falls within a threshold tolerance or a maximum number of iterations is reached.

[0024] The distance metric can compare feature vectors of multiple time samples in two time series to calculate a non-negative scalar value that represents the distance between the two time series.

[0025] The method may further include using at least one hardware processor to perform the process over one or more iterations until a ratio between a number of minority time series in the training data set where a majority of the plurality of labeled time samples include labels representing a minority class and a number of majority time series in the training data set where a majority of the plurality of labeled time samples include labels representing a majority class falls within a threshold tolerance.

[0026] The following list of embodiments provides alternative and / or further features of the present disclosure. 1. A method for reducing class imbalance in a training dataset for machine learning, the method comprising using at least one hardware processor to execute a process, the process comprising: receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset The method is for performing the above-mentioned steps.

[0027] 2. The method of aspect 1, wherein receiving at least a subset of the training dataset includes selecting a random sample of a predetermined number of time series from the training dataset.

[0028] 3. The method of aspect 1, wherein receiving at least a subset of the training dataset includes selecting a sample of a predetermined number of time series from the training dataset such that the training dataset includes a higher proportion of minority time series than non-minority time series, a minority time series being defined as a time series in which a majority of the plurality of labeled time samples include a label that represents a minority class, and a non-minority time series being defined as a time series in which a majority of the plurality of labeled time samples include a label that represents one of a plurality of classes that is not the minority class.

[0029] 4. The method of embodiment 1, wherein the plurality of classes consists of two classes. 5. The method of aspect 1, wherein the number of adjacent time series is a predetermined number greater than one.

[0030] 6. The method of aspect 1, further comprising using at least one hardware processor to generate a predetermined number of synthetic time series for each time series in at least a subset of the training dataset.

[0031] 7. The method of aspect 6, wherein generating the predetermined number of synthetic time series includes randomly selecting one of the identified number of adjacent time series for use in generating the synthetic time series until the predetermined number of synthetic time series is generated.

[0032] 8. For each of one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature generating a composite value of the feature according to a first bias toward the minority class when the labels of corresponding labeled time samples in the time series represent a minority class of the plurality of classes; generating a composite value of the feature according to a second bias that is less biased toward the minority class than the first bias when a label of a corresponding labeled time sample in the time series does not represent the minority class; 2. The method of embodiment 1, comprising:

[0033] 9. The method of embodiment 8, wherein the minority class is defined as one class of the plurality of classes that is less likely to be represented by labels in the plurality of time samples in the plurality of time series in the training dataset than at least one other class of the plurality of classes.

[0034] 10. The method of aspect 9, wherein generating a composite value of the feature according to the first bias includes adding a randomly generated value to a value of the feature in a corresponding labeled time sample in the time series.

[0035] 11. Generating a composite value of a feature according to a second bias, calculating a distance between values ​​of the feature in a corresponding labeled time sample in the time series and values ​​of the feature in the labeled time samples in the adjacent time series; Randomly generating a scaling factor; calculating a scaling value by multiplying the calculated distance by a randomly generated scaling factor; Adding a scaling value to the value of the feature in the corresponding labeled time sample in the time series 10. The method of embodiment 9, comprising:

[0036] 12. The method of aspect 11, wherein when the label of a corresponding labeled time sample in the time series does not represent a minority class and the label of a labeled time sample in an adjacent time series represents a minority class, the range of scaling values ​​is constrained to produce a scaling value that is closer to the value of the feature in the labeled time sample in the adjacent time series than the value of the feature in the corresponding labeled time sample in the time series.

[0037] 13. The method of aspect 11, wherein when the label of the corresponding labeled time sample in the time series does not represent a minority class and the label of the labeled time sample in the adjacent time series does not represent a minority class, the range of scaling values ​​is constrained to produce a scaling value that is closer to the value of the feature in the corresponding labeled time sample in the time series than the value of the feature in the corresponding labeled time sample in the adjacent time series.

[0038] 14. The method of aspect 1, wherein generating the synthetic time series includes labeling the synthetic time series with a label value derived based on a label representing one of a plurality of classes into which the time series is classified and a label representing one of a plurality of classes into which the adjacent time series is classified.

[0039] 15. The method of aspect 14, wherein the label value is further derived based on a distance between the composite time series and the time series and a distance between the composite time series and adjacent time series.

[0040] 16. The method of aspect 15, wherein the plurality of classes consists of two classes, and the label value is derived by binarizing a value based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and an adjacent time series.

[0041] 17. The method of aspect 1, further comprising, after adding the generated synthetic time series to a training dataset, using at least one hardware processor to train a machine learning algorithm using at least a portion of the training dataset.

[0042] 18. The method of aspect 1, wherein the one or more features represent parameters of the electrical system and at least one of the plurality of classes represents an action to be taken in response to an anomaly in the electrical system.

[0043] 19. A system comprising: at least one hardware processor; a memory storing software; Software and the software, when executed by at least one hardware processor, receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset A system configured to:

[0044] 20. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to: receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset A non-transitory computer-readable medium for causing

[0045] 21. A computer-implemented method for detecting an event in an electrical system, the method including using at least one hardware processor to execute a process, the process including: 1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset Reducing class imbalance by adding the generated synthetic time series to a training dataset and then training a machine learning algorithm using at least a portion of the training dataset; executing the trained machine learning algorithm on the acquired data relating to the electrical system; Detecting an event based on execution of a trained machine learning algorithm; 23. A computer-implemented method for performing the steps of:

[0046] 22. The method according to aspect 21, wherein the event is an anomaly in the electrical system, in particular a failure of at least one transmission line in the electrical system.

[0047] 23. The method of aspect 21 or 22, further comprising removing one or more majority time series from the training dataset after adding the generated synthetic time series to the training dataset and before training the machine learning algorithm, wherein a majority of the plurality of labeled time samples include labels that do not represent a minority class.

[0048] 24. The method of embodiment 23, wherein the removal is performed until a maximum number of iterations is reached. 25. The method of aspect 23, wherein the removal is performed until a ratio between the number of minority time series in the training dataset and the number of majority time series in the training dataset is within a threshold tolerance, a minority time series being defined as a time series in which a majority of the plurality of labeled time samples includes a label representing a minority class.

[0049] 26. The method of any of aspects 21-25, wherein receiving at least a subset of the training dataset includes selecting a random sample of a predetermined number of time series from the training dataset.

[0050] 27. A method according to any of aspects 21-26, wherein receiving at least a subset of the training dataset includes selecting a sample of a predetermined number of time series from the training dataset such that the training dataset includes a higher proportion of minority time series than majority time series.

[0051] 28. The method of any of embodiments 21-27, wherein the plurality of classes consists of two classes.

[0052] 29. The method according to any one of aspects 21 to 28, wherein the number of adjacent time series is a predetermined number greater than one.

[0053] 30. The method of any of aspects 21-29, further comprising using at least one hardware processor to generate a predetermined number of synthetic time series for each time series in at least a subset of the training dataset.

[0054] 31. The method of aspect 30, wherein generating the predetermined number of synthetic time series includes randomly selecting one of the identified number of adjacent time series for use in generating the synthetic time series until the predetermined number of synthetic time series is generated.

[0055] 32. For each of one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature includes: generating a composite value of the feature according to a first bias toward the minority class when the labels of corresponding labeled time samples in the time series represent a minority class of the plurality of classes; generating a composite value of the feature according to a second bias that is less biased toward the minority class than the first bias when a label of a corresponding labeled time sample in the time series does not represent the minority class; The method according to any one of aspects 21 to 31, comprising:

[0056] 33. The method of aspect 32, wherein generating a composite value of the feature according to the first bias includes adding a randomly generated value to a value of the feature in a corresponding labeled time sample in the time series.

[0057] 34. Generating a composite value of a feature according to a second bias, calculating a distance between values ​​of the feature in a corresponding labeled time sample in the time series and values ​​of the feature in the labeled time samples in the adjacent time series; Randomly generating a scaling factor; calculating a scaling value by multiplying the calculated distance by a randomly generated scaling factor; Adding a scaling value to the value of the feature in the corresponding labeled time sample in the time series 33. The method of embodiment 32, comprising:

[0058] 35. The method of aspect 34, wherein when the label of a corresponding labeled time sample in the time series does not represent a minority class and the label of a labeled time sample in an adjacent time series represents a minority class, the range of scaling values ​​is constrained to produce a scaling value that is closer to the value of the feature in the labeled time sample in the adjacent time series than the value of the feature in the corresponding labeled time sample in the time series.

[0059] 36. The method of aspect 34, wherein when the label of the corresponding labeled time sample in the time series does not represent a minority class and the label of the labeled time sample in the adjacent time series does not represent a minority class, the range of scaling values ​​is constrained to produce a scaling value that is closer to the value of the feature in the corresponding labeled time sample in the time series than the value of the feature in the corresponding labeled time sample in the adjacent time series.

[0060] 37. A method according to any of aspects 21 to 36, wherein generating the synthetic time series includes labeling the synthetic time series with a label value derived based on a label representing one of a plurality of classes into which the time series is classified and a label representing one of a plurality of classes into which the adjacent time series are classified.

[0061] 38. The method of aspect 37, wherein the label value is further derived based on a distance between the composite time series and the time series and a distance between the composite time series and adjacent time series.

[0062] 39. The method of embodiment 38, wherein the plurality of classes consists of two classes, and the label value is derived by binarizing a value based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and an adjacent time series.

[0063] 40. The method of any of aspects 1-39, wherein the one or more features represent parameters of the electrical system and at least one of the plurality of classes represents an action to be taken in response to an anomaly in the electrical system.

[0064] 41. A system comprising: at least one hardware processor; a memory storing software; Software and the software, when executed by at least one hardware processor, 1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset Reducing class imbalance by adding the generated synthetic time series to a training dataset and then training a machine learning algorithm using at least a portion of the training dataset; executing the trained machine learning algorithm on the acquired data relating to the electrical system; Detecting an event based on execution of a trained machine learning algorithm; A system configured to:

[0065] 42. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to: 1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of a training dataset, the training dataset including a plurality of time series, each of the plurality of time series including a plurality of labeled time samples, each of the plurality of labeled time samples including a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least a subset of the training dataset, identifying a number of adjacent time series in at least a subset of the training dataset based on a distance metric; For each of one or more of the adjacent time series, For each of the one or more features and for each of a plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples including synthetic values ​​of one or more features; Adding the generated synthetic time series to the training dataset Reducing class imbalance by adding the generated synthetic time series to a training dataset and then training a machine learning algorithm using at least a portion of the training dataset; executing the trained machine learning algorithm on the acquired data relating to the electrical system; Detecting an event based on execution of a trained machine learning algorithm; A non-transitory computer-readable medium for causing

[0066] Any of the above methods may be embodied individually or in any combination in executable software modules of a processor-based system, such as a server, and / or in executable instructions stored on a non-transitory computer-readable medium.

[0067] BRIEF DESCRIPTION OF THE DRAWINGS The details of the present disclosure, both as to its structure and operation, can be gleaned in part by study of the accompanying drawings, in which like reference characters refer to like parts, and in which: [Brief description of the drawings]

[0068] [Figure 1] FIG. 1 illustrates an exemplary infrastructure capable of implementing one or more of the processes described herein, according to one embodiment. [Diagram 2] FIG. 1 illustrates an exemplary processing system capable of performing one or more of the processes described herein, according to one embodiment. [Diagram 3] FIG. 1 illustrates a resampling process for reducing class imbalance in a training dataset for machine learning using upsampling, according to one embodiment. [Figure 4] FIG. 1 illustrates a resampling process for reducing class imbalance in a training dataset for machine learning using downsampling, according to one embodiment. [Diagram 5] FIG. 1 illustrates a resampling process to reduce class imbalance in a training dataset for machine learning using both upsampling and downsampling, according to one embodiment. [Figure 6] FIG. 1 illustrates a comparison between precision-recall curves obtained for machine learning models trained without and with a resampled dataset in accordance with an embodiment of the disclosed resampling process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0069] Detailed Description In one embodiment, a system, method, and non-transitory computer-readable medium for reducing class imbalance in a training dataset for machine learning are disclosed. After reading this description, it will be clear to a person skilled in the art how to implement the present disclosure in various alternative embodiments and applications. However, although various embodiments of the present disclosure are described herein, it is understood that these embodiments are presented for the purpose of example and illustration only, and are not limiting. Therefore, this detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure, which is described in the appended claims.

[0070] 1. System Overview 1.1. Infrastructure FIG. 1 illustrates an exemplary infrastructure in which one or more of the disclosed processes may be implemented, according to one embodiment. The infrastructure may comprise a platform 110 (e.g., one or more servers) that hosts and / or executes one or more of the various functions, processes, methods, and / or software modules described herein. The platform 110 may include dedicated servers, or alternatively, may include cloud instances utilizing shared resources of one or more servers. These servers or cloud instances may be collocated and / or geographically distributed. The platform 110 may also comprise or be communicatively connected to a server application 112 and / or one or more databases 114. In addition, the platform 110 may be communicatively connected to one or more user systems 130 via one or more networks 120. The platform 110 may also be communicatively connected to one or more external systems 140 (e.g., other platforms, websites, etc.) via one or more networks 120.

[0071] The network 120 may include the Internet, and the platform 110 may communicate with the user systems 130 over the Internet using standard transmission protocols such as HyperText Transfer Protocol (HTTP), HTTP Secure (HTTPS), File Transfer Protocol (FTP), FTP Secure (FTPS), Secure Shell FTP (SFTP), and proprietary protocols. Although the platform 110 is shown as being connected to various systems through a single set of networks 120, it should be understood that the platform 110 may be connected to various systems through a different set of one or more networks. For example, the platform 110 may be connected to a subset of the user systems 130 and / or external systems 140 over the Internet, but may also be connected to one or more other user systems 130 and / or external systems 140 over an intranet. Further, although only a small number of user systems 130 and external systems 140, one server application 112, and one set of databases 114 are shown, it should be understood that the infrastructure may include any number of user systems, external systems, server applications, and databases.

[0072] User systems 130 may include any one or more types of computing devices capable of wired and / or wireless communication, including, but not limited to, desktop computers, laptop computers, tablet computers, smartphones or other mobile phones, servers, game consoles, televisions, set-top boxes, electronic kiosks, point-of-sale terminals, etc. Each user system 130 may comprise or be communicatively connected to a client application 132 and / or one or more local databases 134. In general, it is contemplated that user systems 130 include personal computers or specialized workstations that a user may utilize to connect to and interact with the platform 110 to resample training datasets as disclosed herein that are stored on the platform (e.g., in database 114) and / or used to train machine learning models hosted on or deployed from the platform 110.

[0073] Platform 110 may comprise a web server that hosts one or more websites and / or web services. In embodiments in which a website is provided, the website may comprise a graphical user interface including one or more screens (e.g., web pages) generated, for example, in HyperText Markup Language (HTML) or other language. Platform 110 transmits or serves one or more screens of the graphical user interface in response to a request from user system 130. In some embodiments, the screens may be served in the form of a wizard, where two or more screens may be served sequentially, with one or more of the sequential screens depending on the interaction of a user or user system 130 with one or more preceding screens. Both requests to platform 110 and responses from platform 110, including screens of the graphical user interface, may be communicated over network 120, which may include the Internet, using standard communication protocols (e.g., HTTP, HTTPS, etc.). These screens (e.g., web pages) can contain a combination of content and elements, such as text, images, video, animations, references (e.g., hyperlinks), frames, inputs (e.g., text boxes, text areas, checkboxes, radio buttons, drop-down menus, buttons, forms, etc.), scripts (e.g., JavaScript), etc., including elements that include or are derived from data stored in one or more databases (e.g., database 114) locally and / or remotely accessible to platform 110. Platform 110 can also respond to other requests from user system 130.

[0074] The platform 110 may include, be communicatively coupled to, or otherwise have access to, one or more databases 114. For example, the platform 110 may include one or more database servers managing the one or more databases 114. Server applications 112 running on the platform 110 and / or client applications 132 running on user systems 130 may submit data (e.g., user data, form data, etc.) stored in the database 114 and / or request access to data stored in the database 114. Any suitable database may be utilized, including, without limitation, MySQL®, Oracle™, IBM®, Microsoft SQL™, Access™, PostgreSQL™, MongoDB™, etc., including cloud-based and proprietary databases. Data may be transmitted to the platform 110, for example, using the well-known POST request supported by HTTP, via FTP, etc. This data and other requests may be handled by server-side web technologies, such as, for example, servlets or other software modules executed by platform 110 (eg, contained in server applications 112).

[0075] In embodiments in which web services are provided, the platform 110 can receive requests from the external systems 140 and provide responses in Extensible Markup Language (XML), JavaScript Object Notation (JSON), and / or any other suitable or desired format. In such embodiments, the platform 110 can provide an application programming interface (API) that defines the manner in which the user systems 130 and / or the external systems 140 can interact with the web services. Thus, the user systems 130 and / or the external systems 140 (which may themselves be servers) can define their own user interfaces and rely on the web services to implement or otherwise provide the back-end processes, methods, functions, storage, etc. described herein. For example, in such embodiments, a client application 132 running on one or more user systems 130 can interact with a server application 112 running on the platform 110 to perform one or more or a portion of one or more of the various functions, processes, methods, and / or software modules described herein. In one embodiment, the client application 132 may utilize a local database 134 to store data locally on the user system 130 .

[0076] The client application 132 may be "thin," in which case processing is primarily performed on the server side by the server application 112 on the platform 110. A basic example of a thin client application 132 is a browser application that simply requests, receives, and renders web pages on the user system 130, while the server application 112 on the platform 110 is responsible for generating the web pages and managing database functions. Alternatively, the client application may be "thick," in which case processing is primarily performed on the client side by the user system 130. It should be understood that the client application 132 may perform an amount of processing for the server application 112 on the platform 110 at any point along this spectrum between "thin" and "thick," depending on the design goals of a particular implementation. In any case, the software described herein may reside entirely on either the platform 110 (e.g., in which case the server application 112 performs all processing) or the user system 130 (e.g., in which case the client application 132 performs all processing), or may be distributed between the platform 110 and the user system 130 (e.g., in which case both the server application 112 and the client application 132 perform processing). The software described herein may include one or more executable software modules that contain instructions for implementing one or more of the processes, methods, or functions described herein.

[0077] 1.2. Exemplary Processing Device 2 is a block diagram illustrating an exemplary wired or wireless system 200 that may be used in connection with various embodiments described herein. For example, system 200 may be used as or in conjunction with one or more of the functions, processes, or methods described herein (e.g., to store and / or execute software) and may represent a component of platform 110, user system 130, external system 140, and / or other processing devices described herein. System 200 may be a server or any conventional personal computer, or any other processor-enabled device capable of wired or wireless data communication. Other computer systems and / or architectures may also be used, as will be apparent to those skilled in the art.

[0078] System 200 preferably includes one or more processors 210. Processor 210 may include a central processing unit (CPU). Additional processors may be provided, such as a graphics processing unit (GPU), auxiliary processors for managing input / output, auxiliary processors for performing floating point operations, dedicated microprocessors (e.g., digital signal processors) having an architecture suitable for fast execution of signal processing algorithms, slave processors (e.g., back-end processors) subordinate to a main processing system, additional microprocessors or controllers for dual or multi-processor systems, and / or co-processors. Such auxiliary processors may be separate processors or may be integrated with processor 210. Examples of processors that may be used with system 200 include, but are not limited to, any of the processors available from Intel Corporation of Santa Clara, California (e.g., Pentium™, Core i7™, Xeon™, etc.), any of the processors available from Advanced Micro Devices, Incorporated (AMD) of Santa Clara, California, any of the processors available from Apple Inc. of Cupertino (e.g., A-series, M-series, etc.), any of the processors available from Samsung Electronics Co., Ltd. of Seoul, Korea (e.g., Exynos™), any of the processors available from NXP Semiconductors NV of Eindhoven, The Netherlands, and the like.

[0079] The processor 210 is preferably connected to a communication bus 205. The communication bus 205 may include a data channel to facilitate information transfer between the storage and other peripheral components of the system 200. Additionally, the communication bus 205 may provide a set of signals used to communicate with the processor 210 and may include a data bus, an address bus, and / or a control bus (not shown). The communication bus 205 may include any standard or non-standard bus architecture, such as, for example, an industry standard architecture (ISA), an extended industry standard architecture (EISA), a Micro Channel Architecture (MCA), a peripheral component interconnect (PCI) local bus, a bus architecture conforming to standards promulgated by the Institute of Electrical and Electronics Engineers (IEEE), including the IEEE 488 general-purpose interface bus (GPIB), IEEE 696 / S-100, and the like.

[0080] The system 200 preferably includes a main memory 215 and may also include a secondary memory 220. The main memory 215 provides storage of instructions and data for programs running on the processor 210, such as any of the software discussed herein. It should be understood that the programs stored in the memory and executed by the processor 210 may be written and / or compiled according to any suitable language, including, but not limited to, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, and the like. Thus, the main memory 215 is typically a semiconductor-based memory, such as a dynamic random access memory (DRAM) or a static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), including read only memory (ROM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), and the like.

[0081] Secondary memory 220 is a non-transitory computer-readable medium on which computer-executable code (e.g., any of the software disclosed herein) and / or other data is stored. Computer software or data stored in secondary memory 220 is read into main memory 215 for execution by processor 210. Secondary memory 220 may include semiconductor-based memory such as, for example, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), and flash memory (a block-oriented memory similar to EEPROM).

[0082] Secondary memory 220 may optionally include internal media 225 and / or removable media 230. Removable media 230 is read from and written to in any known manner. Removable storage medium 230 may be, for example, a magnetic tape drive, a compact disc (CD) drive, a digital versatile disc (DVD) drive, other optical drive, a flash memory drive, or the like.

[0083] In alternative embodiments, secondary memory 220 may include other similar means for allowing computer programs or other data or instructions to be loaded into system 200. Such means may include, for example, a communications interface 240 that allows software and data to be transferred to system 200 from an external storage medium 245. Examples of external storage medium 245 include an external hard disk drive, an external optical drive, an external magneto-optical drive, and the like.

[0084] As mentioned above, the system 200 may include a communications interface 240. The communications interface 240 allows software and data to be transferred between the system 200 and an external device (e.g., a printer), a network, or other information source. For example, computer software or executable code may be transferred to the system 200 from a network server (e.g., the platform 110) via the communications interface 240. Examples of the communications interface 240 include an internal network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a card bus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a wireless data card, a communications port, an infrared interface, an IEEE 1394 Firewire, and any other device that allows the system 200 to interface with a network (e.g., the network 120) or another computing device.Communications interface 240 preferably implements industry published protocol standards such as Ethernet IEEE 802 standard, Fibre Channel, digital subscriber line (DSL), asynchronous digital subscriber line (ADSL), frame relay, asynchronous transfer mode (ATM), integrated digital services network (ISDN), personal communications services (PCS), transmission control protocol / Internet protocol (TCP / IP), serial line Internet protocol / point to point protocol (SLIP / PPP), etc., although customized or non-standard interface protocols may also be implemented.

[0085] The software and data transferred via communications interface 240 is typically in the form of electrical communications signals 255. These signals 255 may be provided to communications interface 240 via communications channel 250. In one embodiment, communications channel 250 may be a wired or wireless network (e.g., network 120), or any of a variety of other communications links. Communications channel 250 carries signals 255 and may be implemented using a variety of wired or wireless communications means including wire or cable, optical fiber, conventional telephone line, a cellular phone link, a wireless data communications link, a radio frequency ("RF") link, or an infrared link, to name just a few.

[0086] Computer-executable code (e.g., computer programs such as the disclosed software) is stored in main memory 215 and / or secondary memory 220. Computer-executable code may also be received via communications interface 240 and stored in main memory 215 and / or secondary memory 220. Such computer programs, when executed, enable system 200 to perform various functions of the disclosed embodiments, as described elsewhere herein.

[0087] In this description, the term "computer-readable medium" is used to refer to any non-transitory computer-readable storage medium used to provide computer-executable code and / or other data to or within system 200. Examples of such media include main memory 215, secondary memory 220 (including internal memory 225, removable media 230, and external storage media 245), and any peripherals communicatively coupled to communication interface 240 (including a network information server or other network device). These non-transitory computer-readable media are means for providing software and / or other data to system 200.

[0088] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into the system 200 by the removable medium 230, the I / O interface 235, or the communication interface 240. In such embodiments, the software is loaded into the system 200 in the form of electrical communication signals 255. The software, when executed by the processor 210, preferably causes the processor 210 to perform one or more of the processes and functions described elsewhere herein.

[0089] In one embodiment, I / O interface 235 provides an interface between one or more components of system 200 and one or more input and / or output devices. Examples of input devices include, but are not limited to, sensors, keyboards, touch screens or other touch-sensitive devices, cameras, biosensing devices, computer mice, trackballs, pen-based pointing devices, etc. Examples of output devices include, but are not limited to, other processing devices, cathode ray tubes (CRTs), plasma displays, light-emitting diode (LED) displays, liquid crystal displays (LCDs), printers, vacuum fluorescent displays (VFDs), surface-conduction electron-emitter displays (SEDs), field emission displays (FEDs), etc. In some cases, input and output devices may be combined, such as in the case of touch-sensitive displays (e.g., in smartphones, tablets, or other mobile devices).

[0090] The system 200 may also include optional wireless communication components that facilitate wireless communication over a voice network and / or a data network (e.g., in the case of user system 130). The wireless communication components include an antenna system 270, a radio system 265, and a baseband system 260. In the system 200, radio frequency (RF) signals are transmitted and received wirelessly by the antenna system 270 under the control of the radio system 265.

[0091] In one embodiment, antenna system 270 may include one or more antennas and one or more multiplexers (not shown) that perform a switching function to provide transmit and receive signal paths for antenna system 270. In the receive path, the received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown), which amplifies the received RF signal and transmits the amplified signal to radio system 265.

[0092] In alternative embodiments, the radio system 265 may comprise one or more radios configured to communicate over various frequencies. In one embodiment, the radio system 265 may combine a demodulator (not shown) and a modulator (not shown) into one integrated circuit (IC). The demodulator and modulator may also be separate components. In the incoming path, the demodulator removes the RF carrier signal, leaving a baseband received audio signal that is transmitted from the radio system 265 to the baseband system 260.

[0093] If the received signal contains audio information, the baseband system 260 decodes the signal and converts it to an analog signal. The signal is then amplified and transmitted to a speaker. The baseband system 260 also receives analog audio signals from a microphone. These analog audio signals are converted to digital signals and encoded by the baseband system 260. The baseband system 260 also encodes the digital signals for transmission and generates a baseband transmit audio signal that is routed to a modulator portion of the radio system 265. The modulator mixes the baseband transmit audio signal with an RF carrier signal and generates an RF transmit signal that can be routed to the antenna system 270 and passed through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 270 where the signal is switched to an antenna port for transmission.

[0094] The baseband system 260 is also communicatively coupled to the processor 210. The processor 210 can access the data storage areas 215 and 220. The processor 210 is preferably configured to execute instructions (i.e., computer programs, such as the disclosed software), which can be stored in the main memory 215 or the secondary memory 220. Computer programs can also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220 or executed upon receipt. Such computer programs, when executed, can enable the system 200 to perform various functions of the disclosed embodiments.

[0095] 2. Process Overview An embodiment of a process for reducing class imbalance in a training dataset for machine learning will now be described in detail. It should be understood that the described process may be embodied in one or more software modules executed by one or more hardware processors (e.g., processor 210), for example, as a software application (e.g., server application 112, client application 132, and / or a distributed application including both server application 112 and client application 132), which may be executed entirely by a processor of platform 110, entirely by a processor of user system 130, or distributed across platform 110 and user system 130, such that some parts or modules of the software application are executed by platform 110 and other parts or modules of the software application are executed by user system 130. The described process may be implemented as instructions expressed in source code, object code, and / or machine code. These instructions may be executed directly by hardware processor 210, or may be executed by a virtual machine operating between the object code and hardware processor 210. In addition, the disclosed software may be built on or interfaced with one or more existing systems.

[0096] Alternatively, the described processes may be implemented as hardware components (e.g., general purpose processors, integrated circuits (ICs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, etc.), a combination of hardware components, or a combination of hardware and software components. To clearly illustrate this interchangeability of hardware and software, various example components, blocks, modules, circuits, and steps are described herein generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functions in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. In addition, the grouping of functions within a component, block, module, circuit, or step is for ease of description. Particular functions or steps may be moved from one component, block, module, circuit, or step to another component, block, module, circuit, or step without departing from the disclosure.

[0097] Further, although the processes described herein are shown with a particular arrangement and order of sub-processes, each process may be implemented with fewer, more, or different sub-processes, as well as with different arrangements and / or orders of the sub-processes. In addition, even if the sub-processes are described or illustrated in a particular order, it should be understood that any sub-process that is not dependent on the completion of another sub-process may be performed before, after, or in parallel with the other independent sub-process.

[0098] 2.1. Introduction Before training a machine learning model such as a classifier, it is important to balance the training dataset. An imbalanced training dataset biases the resulting model towards the majority class because the model is exposed to data that primarily represents the majority class. "Resampling" refers to a data preprocessing method that mitigates imbalances in the training dataset by generating synthetic data that mimics the data in the training dataset. The synthetic data of the minority class modifies the overall distribution of class labels in the training dataset towards a more balanced distribution. One example of a resampling method is the synthetic minority oversampling technique (SMOTE) described by Nitesh et al. in "SMOTE: Synthetic Minority Over-Sampling Technique" (Journal of Artificial Intelligence Research, 16:321-357, 2002), which is incorporated by reference herein as if fully set forth.

[0099] Although resampling has proven effective for independent and identically distributed (IID) data, traditional resampling techniques are often not equally effective for time series data. Time series data includes multiple time samples. Any resampling of time series data should consider the dependency of these time samples along the time dimension. In addition, since time series data has a time dimension, time series data is three-dimensional, resulting in a two-fold class imbalance along the batch or case dimension and the time dimension. In other words, the degree of class imbalance in three-dimensional time series data is more significant than in two-dimensional IID data. For example, the degree of class imbalance in time series data of 50 time samples of each time series is 50 times that of the degree of class imbalance in two-dimensional IID data. Therefore, applying traditional interpolation-based resampling, such as SMOTE, to time series data is very memory intensive and relatively inefficient.

[0100] Accordingly, disclosed herein are embodiments of a process for resampling time series data. In particular, the disclosed embodiments reduce imbalance in a training dataset of time series data in which time samples of the time series are labeled by classes for supervised training of a machine learning classifier. For example, the training dataset can include multiple time series, each of which can include multiple time samples, each of which can include values ​​of one or more features (e.g., a feature vector of feature values) and a label representing the ground truth class into which the time sample was classified. In one embodiment, all of the values ​​of all of the multiple features can be normalized to real values ​​between -1 and 1. As will be appreciated by those skilled in the art, the original dataset can be labeled in any well-known manner (e.g., manual annotation by one or more levels of experts).

[0101] In one embodiment, the resampling process may include an upsampling algorithm that generates synthetic time series data. In alternative or additional embodiments, the resampling process may include a downsampling algorithm that reduces the distribution of the majority class in the existing time series data. It should be understood that an embodiment may consist of only an upsampling algorithm, only a downsampling algorithm, or may include both an upsampling algorithm and a downsampling algorithm. In each embodiment, the goal may be to achieve a ratio of the number of time series belonging to the minority class to the number of majority time series belonging to the majority class in the training dataset that meets or is within a tolerance of a predetermined threshold. It should be understood that for a binomial machine learning classifier (i.e., having only two possible classes), the ideal value for this ratio is 1.0.

[0102] 2.2.Upsampling Algorithm The purpose of upsampling is to reduce the severity of class imbalance in a dataset by synthesizing artificial data that represents the minority class. For time series data, each time sample of the synthetic time series that represents the minority class can be generated from information that is close to the real time series that represents the minority class. Table 1 below shows the pseudocode of the upsampling algorithm according to one embodiment.

[0103] [Table 1-1]

[0104] [Table 1-2]

[0105] 3 illustrates the same upsampling algorithm in a flow chart, according to one embodiment. The upsampling algorithm 300 may be implemented as a software module hosted and / or executed on the platform 110 or user system 130 used to manage training data for machine learning models, train machine learning models, deploy machine learning models, and / or execute machine learning models. The upsampling algorithm 300 may represent an entire resampling process or one module of a comprehensive resampling process.

[0106] The upsampling algorithm 300 can accept one or more parameter values ​​as input. For example, the input to the upsampling algorithm 300 may be an original dataset (e.g., X in Table 1) that represents the original imbalanced training dataset. The upsampling algorithm 300 may also determine the number of time series to sample from the original dataset (e.g., n in Table 1), the number of synthetic time series to be generated from each sampled time series in the original dataset (e.g., n in Table 1), and the number of synthetic time series to be generated from each sampled time series in the original dataset (e.g., n in Table 1). ts ), and the number of neighbors used (e.g., k (k>0) in Table 1). It should be appreciated that these hyperparameters may be selected or calculated based on the degree of class imbalance in the dataset. For example, for larger class imbalance, the number of time series to sample, the number of synthetic time series generated, and / or the number of neighbors used (e.g., n, n in Table 1, respectively) may be selected or calculated based on the degree of class imbalance in the dataset. ts , and / or k) may be increased. The output of the upsampling algorithm 300 is a synthetic dataset (e.g., X in Table 1) that is incorporated into the original dataset. S ) may also be used.

[0107] The upsampling algorithm 300 may also receive or initialize one or more parameters. For example, the upsampling algorithm 300 may identify a minority class in the original dataset (e.g., row 01 of Table 1). It should be understood that each time sample in each time series in the original dataset may be labeled by one of multiple classes. In the case of a binary classifier, there are only two classes, resulting in one class being the minority class and the other class being the majority class. The minority class may be determined as one of the multiple classes for which there is a minimum or smallest number of labeled time samples in the original dataset. In addition, the upsampling algorithm 300 may initialize a data structure for the synthetic dataset (e.g., row 02 of Table 1).

[0108] In sub-process 305 (e.g., corresponding to row 03 in Table 1), a subset (e.g., S in Table 1) of the original dataset (e.g., X in Table 1) may be sampled. All of the time series in the original dataset could be sampled instead of only a smaller subset, but this may be computationally too expensive for a particular system. Thus, in one embodiment, only a smaller subset of the original dataset is sampled. In this case, the time series may be randomly sampled from the original dataset. In one embodiment, the sample is weighted to include a higher proportion of time series belonging to the minority class than the original dataset, such that the sample has a skewed probability distribution to increase the probability of randomly selecting a minority class time series for the original dataset. For a binary classifier, the minority class time series may be defined as a time series that has more minority class time samples than majority class time samples. The subset may be sampled to consist of a predetermined number (e.g., n in Table 1) of time series from the original dataset.

[0109] Sub-process 310 (e.g., corresponding to row 04 in Table 1) may be repeated for each time series in the sampled subset (e.g., S in Table 1) from sub-process 305. If a time series remains to be considered from the sampled subset (i.e., "Yes" in sub-process 310), algorithm 300 proceeds to sub-process 315 to select the next time series TS. i Otherwise, if there are no more time series remaining to be considered from the sampled subset (i.e., “No” at sub-process 310), then algorithm 300 proceeds to sub-process 375.

[0110] In sub-process 315 (e.g., corresponding to row 05 in Table 1), the current time series under consideration, TS i The k nearest neighbors (e.g., K in Table 1) within the sampled subset (e.g., S in Table 1) to the current time series TS are identified. In other words, a predetermined number (e.g., k in Table 1) of nearest neighbors within the sampled subset are identified based on a distance metric. In most cases, the current time series TS i It should be appreciated that while it may be preferred to identify the nearest time series to the current time series T S as the set of neighboring time series, this is not a requirement. In an alternative embodiment, the set of neighboring time series identified in sub-process 315 may be closer to the current time series T S than to time series that are not included in the set of neighboring time series. i For example, the set of adjacent time series may include time series that are far from the current time series TS i may be selected to include neighboring time series that are within a predetermined threshold distance from the current time series TS, but these neighboring time series are more likely to be selected than any neighboring time series that are not included in the set of neighboring time series. i Therefore, whenever the term "k-nearest neighbors" is used herein, it should be understood that alternative embodiments may instead utilize a set of nearby time series (e.g., within a threshold distance) without strictly requiring that it be the set of k-nearest neighbor time series. More generally, the current time series TS iThe neighborhood of may be defined in any manner to include any number of time series according to any criteria and / or metrics.

[0111] Any metric capable of measuring the correlation or similarity between two time series may be used as the distance metric. For example, the distance metric may measure the Euclidean distance between the feature vectors of the time samples of the two time series. As another example, the distance metric may utilize the following correlation distance corr between two vectors x1 and x2 (e.g., feature vectors):

[0112]

number

[0113] In general, in one embodiment, a distance metric accepts as input two time series (e.g., each containing one or more feature vectors) and outputs a scalar value representing the distance between the two time series. The distance metric should be symmetric such that the distance from a first time series to a second time series is the same as the distance from the second time series to the first time series, as calculated by the distance metric. In addition, the output value of the distance metric should be non-negative, and the distance between a time series and itself should be zero.

[0114] If different time series may have different lengths (e.g., different numbers of time samples), the distance metric can utilize dynamic time warping to compare time series of different lengths, for example, as described in "Dynamic Time Warping" in "Information Retrieval for Music and Motion," Springer, Berlin, Heidelberg, doi.org / 10.1007 / 978-3-540-74048-3_4, which is incorporated by reference as if fully set forth herein. It should be understood that dynamic time warping is not necessary when all time series have the same length.

[0115] Sub-process 320 (e.g., corresponding to row 06 in Table 1) generates a predetermined number (e.g., number_synthetic_TS in Table 1) of synthetic time series TS. s The adjacent time series Ts is generated from the k nearest neighbors (e.g., K in Table 1) identified in sub-process 315. n Another synthetic time series can be sampled from the current time series under consideration, TS i If so (i.e., "Yes" in sub-process 320), the algorithm 300 proceeds to sub-process 325 to generate the next time series TS n Instead, we sample the current time series under consideration, TS i If no more synthetic time series are to be generated from (i.e., “No” at sub-process 320 ), then the algorithm 300 returns to sub-process 310 .

[0116] In sub-process 325 (corresponding to, for example, row 07 in Table 1), the adjacent time series TS n is sampled from the k nearest neighbors (e.g., K in Table 1) identified in sub-process 315. In one embodiment, the neighboring time series TS n is randomly selected from the set of k nearest neighbors. However, the adjacent time series TS n It should be understood that may be selected in any different manner.

[0117] In sub-process 330 (e.g., corresponding to row 08 in Table 1), the adjacent time series TS sampled in sub-process 325 is n It is determined whether or not the adjacent time series Ts belongs to a minority class (e.g., the minority in Table 1). n has more minority class time samples than majority class time samples (e.g., adjacent time series TS n If the majority of the time samples in are labeled by the minority class, then the adjacent time series TS n may belong to a minority class. nIf belongs to the minority class (i.e., “Yes” in sub-process 330), the algorithm 300 proceeds to sub-process 335 to s (e.g., initialized in line 09 of Table 1). Instead, n If t does not belong to the minority class (i.e., “No” in sub-process 330), algorithm 300 returns to sub-process 320 to possibly sample a new neighboring time series. In this case, the neighboring time series TS sampled in sub-process 325 is n is effectively discarded.

[0118] Sub-process 335 (e.g., corresponding to row 10 in Table 1) selects one of the features (e.g., the adjacent time series Ts sampled in sub-process 325). n For each feature f in the original dataset, we iterate over the current time series under consideration, TS i and the current adjacent time series TS n It should be understood that the time series TS should have the same set of features, so that both TS and TS have the same feature values. s should also have the same multiple feature values. If feature f remains to be considered from the multiple features (i.e., "Yes" in sub-process 335), algorithm 300 proceeds to sub-process 340. Otherwise, if there are no features remaining to be considered from the multiple features (i.e., "No" in sub-process 335), algorithm 300 proceeds to sub-process 370.

[0119] Sub-process 340 (corresponding, for example, to row 11 in Table 1) processes the adjacent time series TS sampled in sub-process 325. n Alternatively, sub-process 340 may iterate for each time sample t in the current time series TS under consideration. iDepending on how the original dataset is constructed, the time series in the original dataset may or may not all have the same number of time samples t. i and T.S. n If a pair of time samples t do not have the same number of time samples t, subprocess 340 may reorder the time series TS (e.g., to prevent incorrect memory access to a data structure representing the shorter time series). i or T.S. n The number of iterations may be limited to no more than a minimum number of time samples from t. If time sample t remains to be considered (i.e., "Yes" in sub-process 340), algorithm 300 proceeds to sub-process 345. Otherwise, if no time samples remain to be considered (i.e., "No" in sub-process 340), algorithm 300 returns to sub-process 335 to determine whether another feature remains to be considered.

[0120] It should be appreciated that the order of sub-processes 335 and 340 can be easily switched such that algorithm 300 iterates over each feature at each time sample rather than over each time sample of each feature. In either case, sub-processes 335 and 340 collectively iterate over the two-dimensional space of features and time. Overall, the training data set (e.g., X in Table 1) can be thought of as a three-dimensional structure of time series, time samples, and features. Algorithm 300 processes at least a subset of this three-dimensional structure to generate a synthetic time series.

[0121] In sub-process 345 (e.g., corresponding to line 12 of Table 1), the current time series under consideration, TS i (i.e., T.S. i It is determined whether the current feature f of the current time sample t in the current time series TS ([f][t]) is labeled by the minority class. iIf T is labeled with the minority class (i.e., “Yes” in sub-process 345), then algorithm 300 proceeds to sub-process 350. Otherwise, if T is labeled with the minority class (i.e., “Yes” in sub-process 345), then algorithm 300 proceeds to sub-process 350. i If is not labeled by a minority class (i.e., “No” in sub-process 345), algorithm 300 proceeds to sub-process 355. It should be understood that for binary classification, a time series that is not labeled by a minority class is necessarily labeled by a majority class.

[0122] In sub-process 350 (e.g., corresponding to line 13 of Table 1), the synthetic feature time sample TS s The value of [f][t] is the current time series TS under consideration. i Characteristics of time sample TS i In one embodiment of the sub-process 350, the synthetic feature time sample TS can be generated based on the values ​​of [f][t]. s The value of [f][t] is the current time series TS because it is known (i.e., by subprocess 345) to belong to the minority class. i Characteristics of time sample TS i The current adjacent time series under consideration, TS, is generated based on the values ​​of [f][t]. n Characteristics of time sample TS n For example, as shown in Table 1, the synthetic feature time sample TS s The value of [f][t] is the current time series TS, modified based on a random value from a range of possible values. i Characteristics of time sample TS i For example, the random value may be calculated as the value of [f][t] for the current time series TS i Characteristics of time sample TS i It may be added to the value of [f][t].

[0123] In Table 1, the selection of the random values ​​in sub-process 350 is represented by the rdm() function. The rdm() function can return random numbers within a range specified by an input argument, with a uniform distribution (i.e., not biased towards any value in the specified range) over many iterations. For example, rdm(0,0.1) returns a uniform random real value between 0 and 0.1. In particular, in sub-process 350, the range of the random values ​​is set low, resulting in a synthetic feature time sample TS s The resulting value of [f][t] is the current time series TS that belongs to the minority class, as determined in subprocess 345. i Characteristics of time sample TS i [f][t]. It should be understood that the range of 0.0 to 0.1 for the random values ​​is merely an example, and other ranges may be used (e.g., 0.0 to 0.2, 0.1 to 0.3, etc.). In general, the range should be a subset of the values ​​between 0.0 and 0.5, so as to ensure that the random values ​​are closer to 0.0 than to 1.0, but this need not be the case.

[0124] In sub-process 355 (e.g., corresponding to line 14 of Table 1), the current adjacent time series under consideration, TS n (i.e., T.S. n It is determined whether the current feature f of the current time sample t in the current adjacent time series TS ([f][t]) is labeled by the minority class. n If T is labeled with the minority class (i.e., “Yes” in sub-process 355), then algorithm 300 proceeds to sub-process 360. Otherwise, if T is labeled with the minority class (i.e., “Yes” in sub-process 355), then algorithm 300 proceeds to sub-process 360. n If x is not labeled with the minority class (i.e., “No” in sub-process 355, corresponding to row 16 in Table 1), then algorithm 300 proceeds to sub-process 365.

[0125] In sub-process 360 (e.g., corresponding to line 15 of Table 1), the synthetic feature time sample TS s The value of [f][t] is the current time series TS under consideration.i Characteristics of time sample TS i The value of [f][t] and the current adjacent time series TS under consideration n Characteristics of time sample TS n For example, the synthetic feature time sample TS can be generated based on both the values ​​of [f][t] and [t] as shown in Table 1. s The value of [f][t] is the current time series TS i Characteristics of time sample TS i For the value of [f][t], we set the current adjacent time series TS to a random value from a range of possible values. n Characteristics of time sample TS n The value of [f][t] and the current time series TS i Characteristics of time sample TS i It may be calculated as the sum of the products of multiplication of the difference between the values ​​of [f][t].

[0126] In Table 1, the selection of the random value in sub-process 360 is again represented by the rdm() function. In particular, in sub-process 360, the range of the random value is set high, so that the synthetic feature time sample TS s The resulting value of [f][t] is the current time series TS i Characteristics of time sample TS i The value of [f][t] is determined in subprocess 355 as the current adjacent time series TS that belongs to the minority class. n Characteristics of time sample TS n For example, in Table 1, the range of random values ​​is set to 0.9 to 1.0, which is close to the value of [f][t] for the feature time sample TS i The value of [f][t] and the characteristic time sample TS n Within the range of possible values ​​between the value of [f][t], the synthetic feature time sample TS s The obtained value of [f][t] is the characteristic time sample TS n This means that [f][t] is guaranteed to be within the top 10% of the closest values. As another example, if the range of the random values ​​is set to 0.7 to 0.9, the synthetic feature time sample TS s The resulting value of [f][t] is the characteristic time sample TSn [f][t] is not within the top 10% of the closest values, but the characteristic time sample TS n The random value is guaranteed to be within the top 30% of the closest values ​​to [f][t]. In other words, the random value is the characteristic time sample TS i The value of [f][t] and the adjacent feature time samples TS n [f][t] acts as a scaling factor for downscaling the distance between. It should be understood that the 0.9 to 1.0 range of random values ​​is merely an example, and other ranges may be used (e.g., 0.8 to 1.0, 0.7 to 0.9, etc.). In general, the range should be a subset of the values ​​between 0.5 and 1.0, so as to ensure that the random values ​​are closer to 1.0 than to 0.0, but this need not be the case.

[0127] In sub-process 365 (e.g., corresponding to line 17 of Table 1), the synthetic feature time sample TS s The value of [f][t] is the current time series TS under consideration. i Characteristics of time sample TS i The value of [f][t] and the current adjacent time series TS under consideration n Characteristics of time sample TS n For example, the synthetic feature time sample TS can be generated based on both the values ​​of [f][t] and [t] as shown in Table 1. s The value of [f][t] is the current time series TS i Characteristics of time sample TS i For the value of [f][t], we set the current adjacent time series TS to a random value from a range of possible values. n Characteristics of time sample TS n The value of [f][t] and the current time series TS i Characteristics of time sample TS i It may be calculated as the sum of the products of multiplication of the difference between the values ​​of [f][t].

[0128] In Table 1, the selection of the random value in sub-process 365 is again represented by the rdm() function. However, in contrast to sub-process 360, in sub-process 365 the range of the random value is set lower, resulting in a synthetic feature time sample TS s The obtained value of [f][t] is the current adjacent time series TS n Characteristics of time sample TS n The value of [f][t] is less than the current time series TS i Characteristics of time sample TS i For example, in Table 1, the range of the random value is set to 0.0 to 0.1, which is close to the value of [f][t] for the feature time sample TS i The value of [f][t] and the characteristic time sample TS n Within the range of possible values ​​between the value of [f][t], the synthetic feature time sample TS s The obtained value of [f][t] is the characteristic time sample TS i This means that the random value is guaranteed to be within the top 10% of values ​​closest to [f][t]. It should be understood that the range of 0.0 to 0.1 for the random values ​​is just an example, and other ranges may be used (e.g., 0.0 to 0.2, 0.1 to 0.3, etc.). In general, the range should be a subset of the values ​​between 0.0 and 0.5, so that the random value is guaranteed to be closer to 0.0 than to 1.0, but this need not be the case.

[0129] It should be understood that the generation of the composite values ​​described with respect to sub-processes 350, 360, and 365 is merely an example, and the composite values ​​may be generated in other manners. For example, in an alternative embodiment, the composite values ​​are all generated based on the current time series TS i Characteristics of time sample TS i The characteristic time sample TS may be generated by adding a random number to [f][t]. i When [f][t] is in the minority class, the random number is the feature time sample TS i Alternatively, the synthesis value may be generated according to a threshold. For example, the feature time sample TS may be generated such that [f][t] is smaller than if it were not in the minority class. iWhen [f][t] is in the minority class, the composite value is the feature time sample TS i [f][t] within a first threshold distance and / or the current neighboring time series TS n Characteristics of time sample TS n [f][t] may be randomly generated to be at least a second threshold distance outward from the characteristic time sample TS. i [f][t] is not in the minority class, and the feature time sample TS n When [f][t] is in the minority class, the composite value is the feature time sample TS n [f][t] within a first threshold distance from and / or characteristic time sample TS i [f][t] may be randomly generated to be at least a second threshold distance outward from [f][t]. More generally, the characteristic time samples TS i When [f][t] is in the minority class, the composite value may be generated according to a first bias towards the minority class, and the feature time sample TS i When [f][t] is not in the minority class, the composite value may be generated according to a second bias that is less biased toward the minority class than the first bias (e.g., biased away from the minority class, biased toward the majority class, biased toward the minority class but to a lesser extent than the first bias, etc.).

[0130] In sub-process 370 (e.g., corresponding to line 22 of Table 1), a new synthetic time series TS s to a synthetic dataset (e.g., X in Table 1). s ) or a new synthetic time series TS s may be added directly to the original dataset (e.g., X in Table 1). Subprocess 370 adds a label to the synthetic time series TS, representing the class it belongs to. s (e.g., corresponding to line 21 of Table 1). In one embodiment, the label is added to the synthetic time series TS s , and the synthetic time series TS s The time series TS was created i and adjacent time series TSn For example, in Table 1, for a binary class, the synthetic time series T n The label of the current time series TS i Label the synthetic time series TS s and the current adjacent time series TS n and the product of the distance between the current adjacent time series TS n Label the synthetic time series TS s and the current time series TS i The sum of the product of multiplying the distance between s and the current time series TS i and the distance between the synthetic time series TS s and the current adjacent time series TS n The distance between the time series may be calculated using any distance metric, including the same distance metric used to identify the k nearest neighbors in subprocess 315 described above. i and T.S. n It should be understood that the value of a given label in can be either 0 or 1. However, in the synthetic time series TS s The label values ​​of the synthetic time series TS s may be a real value representing the probability of the class to be classified. In alternative embodiments, the values ​​of the labels in a binary classification may be binarized to either the minority class or the majority class (e.g., by rounding the real value to produce either 0 or 1). In such embodiments, the following line may be added after line 21 of Table 1 to binarize the class labels using a binarization threshold (e.g., binarization_threshold), which may be user-specified (e.g., included as an input to algorithm 300) or system-specified:

[0131] [Table 2]

[0132] In sub-process 375 (e.g., corresponding to line 27 of Table 1), a synthetic data set (e.g., X s ) is output as a product of the algorithm 300. In an alternative embodiment, the synthetic dataset may be directly incorporated into the original dataset.

[0133] In summary, the upsampling algorithm 300 may sample time series from the original dataset and then iteratively pair each time series with its k nearest neighbors that belong to the minority class to create a synthetic time series. The use of k nearest neighbors and distance metrics concerns the temporal correlation between the time samples to ensure that the synthesized time samples are not too far from the original samples. For each feature of each time sample, a synthetic feature value of the synthetic time sample is generated. If possible, each synthetic feature value is calculated to be close to the feature value of the time sample that belongs to the minority class. In other words, the synthetic feature value mimics the feature value of the time sample that belongs to the minority class. As a result, the synthetic time series created from these synthetic feature values ​​mimics the time series that belong to the minority class. These synthetic time series may then be incorporated into the training dataset to increase the ratio of the number of time series that belong to the minority class to the total number of time series and / or the number of time series that belong to the majority class. In particular, the upsampling algorithm 300 can reuse already available time series data such that new time series data does not need to be acquired, and can achieve the desired ratio while preserving the entire original data set (e.g., not downsampled).

[0134] 2.3.Downsampling Algorithm The purpose of downsampling is to reduce the severity of class imbalance in a dataset by removing data that represents the majority class. In the case of time series data, a subset of the time series that belong to the majority class can be removed from the dataset, thereby adjusting the ratio of the number of time series that belong to the minority class to the number of time series that belong to the majority class.

[0135] 4 illustrates a downsampling algorithm, according to one embodiment. The downsampling algorithm 400 may be implemented as a software module hosted and / or executed on the platform 110 or user system 130 used to manage training data for machine learning models, train machine learning models, deploy machine learning models, and / or execute machine learning models. The downsampling algorithm 400 may represent one module of an entire resampling algorithm or a comprehensive resampling algorithm (e.g., combined with the upsampling algorithm 300).

[0136] In sub-process 410, algorithm 400 may receive the data set to be downsampled. Additionally, algorithm 400 may receive one or more hyperparameters, such as a threshold and a tolerance. The threshold represents an ideal or desired ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class. In one embodiment, the threshold is 1.0. The tolerance represents an acceptable difference between the actual ratio and the desired ratio. In other words, the objective of algorithm 400 is to adjust the data set such that the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class is greater than the threshold minus the tolerance and less than the threshold plus the tolerance. As an example, the tolerance may be 0.01. In general, the tolerance may be any scalar value appropriate for a given task, and will typically be in the range of 0.001 to 0.01.

[0137] In sub-process 420, it can be determined whether the absolute value of the difference between the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class and a threshold is less than a tolerance value. A time series may belong to a minority class when a majority of its time samples are labeled by the minority class. Similarly, a time series may belong to a majority class when a majority of its time samples are labeled by the majority class. If this absolute value is greater than or equal to the tolerance value (i.e., "No" in sub-process 420), algorithm 400 proceeds to sub-process 430. Otherwise, if this absolute value is less than the tolerance value (i.e., "Yes" in sub-process 420), algorithm 400 proceeds to sub-process 450.

[0138] In sub-process 430, multiple-class time series are sampled from the dataset. In one embodiment, the sampled time series are randomly selected from the set of multiple-class time series in the dataset. In addition, the dataset may be shuffled before selecting the time series. However, it should be understood that the time series may be selected in any different manner.

[0139] In sub-process 440, the sampled time series may be removed from the data set. Algorithm 400 may then return to sub-process 420 to re-evaluate the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class. In an alternative embodiment, sub-process 430 may sample the number of time series that need to be removed to bring the ratio within tolerance from the threshold, and sub-process 440 may remove all of the sampled time series at once, such that sub-processes 420-440 only need to be performed once.

[0140] In sub-process 450, a dataset may be output. It should be appreciated that the output dataset may be reduced in size by the number of time series removed in one or more iterations of sub-process 440. As a result of removing those time series that belong to the majority class, the output dataset may be more balanced than the input dataset. For example, in the case of binomial classes, the ratio of the number of time series that belong to the minority class to the number of time series that belong to the majority class in the output dataset will be closer to 1.0 than in the input dataset.

[0141] 2.4. Exemplary Resampling Process In one embodiment, the resampling process includes both the upsampling algorithm 300 and the downsampling algorithm 400. Alternatively, the resampling process may include the upsampling algorithm 300, but not the downsampling algorithm 400. As yet another alternative, the resampling process may include the downsampling algorithm 400, but not the upsampling algorithm 300.

[0142] 5 illustrates a resampling process for reducing class imbalance in a training dataset for machine learning using both upsampling and downsampling, according to one embodiment. Process 500 may be implemented as a software module hosted and / or executed on platform 110 or user system 130 used to manage training data for machine learning models, train machine learning models, deploy machine learning models, and / or run machine learning models.

[0143] In sub-process 510, process 500 may receive a dataset to be resampled. In addition, process 500 may receive one or more hyperparameters, such as any of the hyperparameters described with respect to algorithms 300 and 400. For example, these hyperparameters may include the number of time series to sample from the original dataset, the number of synthetic time series to be generated from each sampled time series in the original dataset, and the number of nearest neighbors to be used, as described with respect to algorithm 300. These hyperparameters may also include thresholds and tolerances, as described with respect to algorithm 400.

[0144] In sub-process 520, it may be determined whether the absolute value of the difference between the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class in the data set received in sub-process 510 and a threshold value is less than a tolerance value. Sub-process 520 may be the same as or similar to sub-process 420, and therefore any description of sub-process 420 applies equally to sub-process 520 and vice versa. If this absolute value is greater than or equal to the tolerance value (i.e., "No" in sub-process 520), process 500 may execute upsampling algorithm 300. Otherwise, if this absolute value is less than the tolerance value (i.e., "Yes" in sub-process 520), process 500 may proceed to sub-process 550.

[0145] As described elsewhere herein, the upsampling algorithm 300 can generate a synthetic time series based on the data set received in sub-process 510. Essentially, the upsampling algorithm 300 generates a synthetic time series by calculating average information from adjacent time series, thereby reducing temporal bias through randomization and convex combination techniques. The upsampling algorithm 300 ensures that there are enough time series representing minority classes to properly balance a highly imbalanced data set.

[0146] In sub-process 530, the synthetic dataset including the synthetic time series generated by the upsampling algorithm 300 may be incorporated into the original dataset to generate an extended dataset. In an alternative embodiment, the upsampling algorithm 300 may incorporate the synthetic time series into the original dataset as it is generated to directly generate the extended dataset such that sub-process 530 may be omitted.

[0147] To correct for any residual class imbalance, process 300 can input the expanded data set into a downsampling algorithm 400. As described elsewhere herein, downsampling algorithm 400 can remove time series that belong to the majority class until the ratio of the number of time series that belong to the minority class to the number of time series that belong to the majority class is within a threshold tolerance. Downsampling algorithm 400 provides precise control over this ratio to balance the expanded data set to a predetermined accuracy represented by a tolerance.

[0148] It should be appreciated that if the upsampling algorithm 300 produces a number of synthetic time series that increase the ratio to within the threshold tolerance, then the downsampling algorithm 400 can terminate without removing any time series that belong to the majority class. In other words, if the first iteration of sub-process 420 results in a "Yes" decision, then the algorithm 400 performs sub-processes 410, 420, and 450 without performing any iterations of sub-processes 430 and 440.

[0149] In sub-process 550, the expanded and possibly downsampled dataset is output. It should be appreciated that the output dataset has been rebalanced to within an acceptable level of class imbalance. This output dataset can then be used to train a machine learning model, such as a binary classifier. As a result of a more balanced training dataset, the resulting machine learning model can more accurately identify time series that belong to minority classes, which may represent rare occurrences in mission-critical applications.

[0150] 3. Example Use Cases It should be understood that the processes described herein, including algorithm 300, algorithm 400, and resampling process 500, may be used to resample a dataset for any machine learning model classifying any type of time series in any application or context. However, one particular application is described in detail to provide a concrete, non-limiting example that demonstrates the capabilities of the disclosed processes. It should be understood that similar results can be achieved in other applications using the disclosed embodiments.

[0151] For demonstration purposes, a training dataset for a machine learning algorithm for time domain protection was rebalanced using the resampling process 500. Time domain protection represents a mission-critical application in power systems. In time domain protection, the task is to train a machine learning model to classify features of the electric system for fault detection, phase selection, reach, etc. The training dataset may be unbalanced due to the problem definition. For example, an overhead line fault may have a very high or very low reach setting. Assuming a uniform distribution of fault locations in the original training dataset, a reach setting of 10% would result in only 10% of all time series in the original training dataset being labeled by a fault class. It should be understood that in this case, the fault or trip class (e.g., representing a fault that warrants tripping of a circuit breaker) is the minority class. The majority class may be a non-fault or suppression class (e.g., representing a fault that does not represent a fault or a fault that does not warrant tripping of a circuit breaker). Alternatively, for a high reach setting, the fault or trip class may be the majority class, while the non-fault or suppression class may be the minority class. It should therefore be appreciated that the reach setting generally determines which classes are majority classes and which classes are minority classes in the training dataset.

[0152] Unlike IID data, time series data is three-dimensional with time samples that are correlated along the time dimension. As a result, class imbalance is magnified in time series data, which makes traditional resampling techniques such as bootstrapping very expensive in terms of processing time and memory requirements. In contrast to these traditional resampling techniques, the disclosed resampling embodiments are consistent across correlated time samples, are efficient in achieving rebalancing within a given tolerance that may vary depending on the particular application, and preserve the temporal correlation of the time samples by considering the entire sequence of features in the time series when imputing a synthetic time series during upsampling.

[0153] In addition, the disclosed embodiments facilitate the application of machine learning to contexts with variable settings. For example, in the context of time domain protection, a machine learning model may be trained to classify time series into one of two classes that define either a tripping decision (e.g., trip a circuit breaker) or a suppression decision (e.g., do not trip a circuit breaker). This decision is influenced by a given set of reach settings, making class prediction and resulting decisions more difficult than in contexts with fixed reach settings.

[0154] As defined below, precision and recall are two metrics that can be used to capture the strength of a trained machine learning model.

[0155]

number

[0156] In the case of class imbalance in the training dataset, we may give unequal importance to false positives and false negatives. Therefore, a good candidate for capturing the predictive accuracy of a machine learning classifier is F β score, which is also known as the F1 score when β=1. β The score is a generalization of the harmonic mean of precision and recall and is given by:

[0157]

number

[0158] During experiments with the disclosed resampling technique, an attention mechanism was used to capture the relative importance of past information in current training. The particular attention mechanism used is described in Thang et al., "Effective Approaches to Attention-based Neural Machine Translation," arXiv abs / 1508.04025 (2015), which is incorporated by reference as if fully set forth herein. Different machine learning models were trained with acausal attention (i.e., all time samples were used to learn attention weights), causal attention (i.e., future time samples were masked while calculating attention weights at a time point, and all time samples up to this time point were used to calculate this attention weight), and no attention mechanism. Multiple machine learning models were also trained with different hyperparameters, with and without the disclosed resampling technique.

[0159] The experiments demonstrated that the predictive accuracy of the machine learning models, as measured by the F1 score, is improved when trained on datasets that have been resampled to reduce class imbalance according to the disclosed resampling technique. In particular, three machine learning models were trained for fault detection in time domain protection: (1) a fixed line length of 200 kilometers with a reach setting within 130-190 kilometers, (2) a variable line length with a reach setting within 120-180 kilometers, and (3) a variable line length with a reach setting within 50-120 kilometers. For comparison, each machine learning model was trained independently on both a non-resampled training dataset and a training dataset that was resampled according to the disclosed resampling technique. The training datasets included time series created from a one tower line with a fixed line length and time series created from a two tower line with different line lengths. The F1 scores of the trained machine learning models are shown in Table 2 below.

[0160] [Table 3]

[0161] 6 shows a comparison between precision-recall curves obtained for machine learning models trained separately without and with a resampled dataset according to the disclosed resampling technique. In particular, without the disclosed resampling of the training dataset, the accuracy of the obtained machine learning model was significantly lower, with relatively more false positives.

[0162] As demonstrated by the above results, the disclosed resampling technique for reducing class imbalance in time series data can improve machine learning classification of rare events. In power systems, rare events may occur in the form of transmission line faults. Thus, the disclosed resampling technique can be used to improve the forecast accuracy of distance protection and time domain protection.

[0163] Apart from protection tasks, class imbalance in time series data may occur in cascading faults in power grids, where rare combinations of conditions, uncommon events, and / or inappropriate countermeasures may result in further line breaks and ultimately lead to fault propagation. In such situations, balancing the class labels in the training dataset may result in better estimation and, consequently, prevention of cascading faults, which are rare but highly impactful events. Another context in which the disclosed resampling technique may be particularly useful is load forecasting from time series data. In load forecasting, rare load events such as lightning, heat waves, etc. may create an imbalanced dataset. In such cases, the dataset may be resampled by the disclosed process before training a machine learning model to perform load forecasting. Other potential contexts include resampling of training datasets for machine learning models classifying time series data into classes for power outage detection, congestion, circuit breaker status (e.g., open or closed), inrush operation status for normal or rare events (e.g., external fault, internal fault such as bushing, or explosion), transformer status (e.g., inrush, normal, internal fault, or external fault ride-through), line fault status (e.g., AG, BG, CG, AB, BC, or CA), anomalies or outliers in Supervisory Control and Data Acquisition (SCADA) data, etc. In general, any time series dataset can be resampled according to the disclosed resampling techniques, especially for binary classification.

[0164] Another exemplary situation in which resampling may be particularly useful is the estimation of the probability of contingencies in a power grid. The topology of a transmission substation may change as a result of actions taken by operators to maintain equipment and mitigate undesirable operating conditions such as high circuit loads, out-of-limit voltages, forced outages, and the like. In some cases, the variation in topology may result in a situation in which a protection system, in response to the occurrence of a fault on the network, operates to remove two or more elements when clearing the fault. Such topologies significantly increase the risk level of the network because they expose the network to multiple fault contingencies as a result of a single fault, and because the risk associated with a particular contingency is the expected value of the outcome of the contingency. Thus, estimating the risk of higher order contingencies is important for the safety of the power system, and there is a need to estimate the probability of contingencies. Thus, the disclosed resampling technique can be applied to time series data used to train machine learning models to estimate these probabilities of contingencies.

[0165] The above description of the disclosed embodiments is provided to enable those skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. It should therefore be understood that the description and drawings presented herein represent the presently preferred embodiments of the present disclosure, and thus represent the subject matter broadly contemplated by the present disclosure. It is further understood that the scope of the present disclosure fully encompasses other embodiments that may become apparent to those skilled in the art, and thus the scope of the present disclosure is not limited.

[0166] Combinations described herein such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, and any such combination may include one or more members of its components A, B, and / or C. For example, a combination of A and B may include one A and multiple Bs, multiple As and one B, or multiple As and multiple Bs.

Claims

1. 1. A computer-implemented method for detecting an event in an electrical system, the method comprising using at least one hardware processor to execute a process, the process comprising:

1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of the training dataset, the training dataset comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least the subset of the training data set, identifying a number of adjacent time series in at least the subset of the training data set based on a distance metric; For each of one or more of the adjacent time series, for each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples comprising the synthetic values ​​of the one or more features; adding the generated synthetic time series to the training dataset; and Reducing class imbalance by training a machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset; executing the trained machine learning algorithm on acquired data relating to the electrical system; detecting the event based on the execution of the trained machine learning algorithm; 23. A computer-implemented method for performing the steps of:

2. The method according to claim 1 , wherein the event is an abnormality in the electrical system, in particular a failure of at least one transmission line of the electrical system.

3. 3. The method of claim 1 or 2, further comprising removing one or more majority time series from the training dataset after adding the generated synthetic time series to the training dataset and before training the machine learning algorithm, wherein a majority of the plurality of labeled time samples include labels that do not represent the minority class.

4. The method of claim 3 , wherein the removal is performed until a maximum number of iterations is reached.

5. 4. The method of claim 3, wherein the pruning is performed until a ratio between a number of minority time series in the training dataset and a number of majority time series in the training dataset is within a threshold tolerance, a minority time series being defined as a time series in which a majority of the plurality of labeled time samples includes a label representing the minority class.

6. The method of any of claims 1 to 5, wherein receiving at least a subset of a training dataset comprises selecting a random sample of a predetermined number of time series from the training dataset.

7. 7. The method of claim 1, wherein receiving at least a subset of a training dataset comprises selecting a sample of a predetermined number of time series from the training dataset such that the training dataset includes a higher proportion of minority time series than majority time series.

8. The method of any preceding claim, wherein the plurality of classes consists of two classes.

9. The method according to any one of claims 1 to 8, wherein the number of adjacent time series is a predetermined number greater than one.

10. 10. The method of claim 1, further comprising using the at least one hardware processor to generate a predetermined number of synthetic time series for each time series in the at least the subset of the training data set.

11. 11. The method of claim 10, wherein generating a predetermined number of synthetic time series comprises randomly selecting one of the identified number of adjacent time series for use in generating a synthetic time series until the predetermined number of synthetic time series are generated.

12. For each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature includes: generating the composite value of the feature according to a first bias towards a minority class when the label of a corresponding labeled time sample in the time series represents a minority class of the plurality of classes; generating the composite value of the feature according to a second bias that is less biased toward the minority class than the first bias when the label of the corresponding labeled time sample in the time series does not represent the minority class; The method according to any one of claims 1 to 11, comprising:

13. 13. The method of claim 12, wherein generating the composite value of the feature according to the first bias comprises adding a randomly generated value to the value of the feature in the corresponding labeled time sample in the time series.

14. Generating the composite value of the feature according to the second bias comprises: calculating a distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series; Randomly generating a scaling factor; calculating a scaling value by multiplying the calculated distance by the randomly generated scaling factor; adding the scaling value to the value of the feature in the corresponding labeled time sample in the time series; The method of claim 12, comprising:

15. 15. The method of claim 14, wherein when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series represents the minority class, the range of scaling values ​​is constrained to produce scaling values ​​that are closer to the value of the feature in the labeled time sample in the adjacent time series than the value of the feature in the corresponding labeled time sample in the time series.

16. 15. The method of claim 14, wherein when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series does not represent the minority class, the range of scaling values ​​is constrained to produce a scaling value that is closer to the value of the feature in the corresponding labeled time sample in the time series than the value of the feature in the corresponding labeled time sample in the adjacent time series.

17. 17. The method of claim 1, wherein generating the synthetic time series comprises labeling the synthetic time series with a label value derived based on a label representing one of the classes into which the time series is classified and a label representing one of the classes into which the adjacent time series are classified.

18. The method of claim 17 , wherein the label value is further derived based on a distance between the composite time series and the time series and a distance between the composite time series and the adjacent time series.

19. 20. The method of claim 18, wherein the plurality of classes consists of two classes, and the label values ​​are derived by binarizing a value based on the distance between the composite time series and the time series and the distance between the composite time series and the adjacent time series.

20. The method of any preceding claim, wherein the one or more features represent parameters of an electrical system and at least one of the plurality of classes represents an action to be taken in response to an anomaly in the electrical system.

21. 1. A system comprising: at least one hardware processor; a memory storing software; The software the software, when executed by the at least one hardware processor, 1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of the training dataset, the training dataset comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least the subset of the training data set, identifying a number of adjacent time series in at least the subset of the training data set based on a distance metric; For each of one or more of the adjacent time series, for each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples comprising the synthetic values ​​of the one or more features; adding the generated synthetic time series to the training dataset; and Reducing class imbalance by training the machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset; executing the trained machine learning algorithm on acquired data relating to the electrical system; detecting the event based on the execution of the trained machine learning algorithm; A system configured to:

22. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:

1. Reducing class imbalance in a training dataset for a machine learning algorithm, comprising: receiving at least a subset of the training dataset, the training dataset comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time point within a time frame of the time series and a label representing one of a plurality of classes into which the time sample was classified; For each time series in at least the subset of the training data set, identifying a number of adjacent time series in at least the subset of the training data set based on a distance metric; For each of one or more of the adjacent time series, for each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a composite value of the feature in a composite time sample; generating a synthetic time series from the synthetic time samples comprising the synthetic values ​​of the one or more features; adding the generated synthetic time series to the training dataset; and Reducing class imbalance by training the machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset; executing the trained machine learning algorithm on acquired data relating to the electrical system; detecting the event based on the execution of the trained machine learning algorithm; A non-transitory computer-readable medium for causing

Images