Reducing category imbalance in machine learning training datasets
By generating synthetic time series and adjusting the training data set, the problem of category imbalance in machine learning models is solved, and the accuracy and performance of the model is improved, especially in power system fault detection, and better rare event recognition capabilities are achieved. .
Patent Information
- Application Number
- CN202380030562.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-12
- Filing Date
- 2023-03-10
- Publication Date
- 2024-11-15
AI Technical Summary
The problem of category imbalance in machine learning models in training data sets leads to performance degradation, especially in rare event recognition applications, such as power system fault detection. The existing technology is difficult to effectively solve category imbalance, affecting the accuracy of the model.
By generating synthetic time series of a few categories and reducing the number of time series of most categories, using distance metrics to identify adjacent time series, generating synthetic time series, adding to the training dataset, adjusting the dataset to achieve category ratio balance, and training machine learning algorithms to Improve the accuracy of the model.
By balancing the training dataset, the accuracy and performance of machine learning models are improved, especially in identifying rare events, and the predictive ability of power system failure detection is enhanced.
Smart Images

Figure CN118974747A_ABST
Abstract
Description
Technical Field
[0001] Embodiments described herein are directed generally to machine learning, and more particularly, to reducing class imbalance in datasets used to train machine learning models. Background Art
[0002] In many mission-critical applications, rare events must be correctly identified so that appropriate actions can be taken. Protection of power systems is one such application. In particular, when a fault occurs in a power system, a protection device may be used to disconnect the faulty section of the power system from the rest of the power system in order to contain the fault. Alternatively, a monitoring device may be used to identify abnormal behavior in the power system. It should be understood that in other types of mission-critical applications, there are similar rare events for which responsive actions must be selected and taken. Machine learning is a useful tool for responding to events in systems (including mission-critical applications). Machine learning models are trained to produce outputs (e.g., classifications) based on input features. Since machine learning is data-driven, the quality of the data set used to train the model will become a key factor in the accuracy of the resulting model. In supervised learning of a machine learning classifier, the training data set includes multiple sets of feature values (e.g., feature vectors) that are labeled with their corresponding true categories.
[0003] If the possible classes include a relatively rare occurring class, the label distribution in the training dataset will be severely skewed away from the rare class and towards (multiple) other classes. In other words, the rare class may be severely underrepresented in the training dataset, making the training dataset unbalanced. This imbalance may adversely affect the performance of any machine learning model trained on the dataset. Summary of the invention
[0004] Thus, systems, methods, and non-transitory computer-readable media for reducing class imbalance in a training dataset for machine learning are disclosed. An object of an embodiment is to increase the proportion of time series of a minority class in a training dataset by generating a synthetic time series of the minority class and / or reducing the number of time series of the majority class, thereby reducing the class imbalance in the training dataset (e.g., within a tolerance). A further object of some embodiments is to generate a synthetic time series of the minority class using neighboring time series for each of the samples of the time series in the training dataset, for example based on the distance between time samples in a pair of time series. A further object of some embodiments is to use the training dataset to train a machine learning algorithm, such as a binary classifier, for which the class imbalance has been reduced by the disclosed techniques. Advantageously, the reduction in class imbalance in the training dataset produces greater accuracy in a machine learning classifier trained with a more balanced training dataset.
[0005] In an embodiment, a method for reducing class imbalance in a training dataset for machine learning is disclosed, the method comprising using at least one hardware processor to execute a process to: receive at least a subset of the training dataset, the training dataset comprising multiple time series, each of the multiple time series comprising multiple labeled time samples, each of the multiple labeled time samples comprising a value of each of one or more features (e.g., multiple features) at a specific time within a time frame of the time series, and a label indicating that the time sample has been classified into one of multiple categories; for each time series in the at least subset of the training dataset: identify several neighboring time series in the at least subset of the training dataset based on a distance metric, for each of one or more of the neighboring time series, for each of the multiple labeled time samples in the neighboring time series, and for each of the one or more features, generate a synthetic value of the feature in a synthetic time sample; and generate a synthetic time series based on the synthetic time samples comprising the synthetic values of the one or more features; and add the generated synthetic time series to the training dataset.
[0006] In a further embodiment, a method for detecting an event in an electrical system is disclosed, the method comprising using at least one hardware processor to perform a process to: reduce class imbalance in a training data set for a machine learning algorithm by: receiving at least a subset of the training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a specific time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories; for each time series in the at least subset of the training data set: identifying the at least one feature of the training data set based on a distance metric; a plurality of adjacent time series in a subset; for each of one or more of the adjacent time series: for each of the one or more features and for each of the plurality of labeled time samples in the adjacent time series, generating a synthetic value of the feature in a synthetic time sample; and generating a synthetic time series of synthetic values based on the synthetic time samples including the one or more features; and adding the generated synthetic time series to a training data set; after adding the generated synthetic time series to the training data set, using at least a portion of the training data set to train a machine learning algorithm; executing the trained machine learning algorithm on the obtained data related to the electrical system; and detecting the event based on the execution of the trained machine learning algorithm.
[0007] The event may be an anomaly in the electrical system, in particular a fault in at least one transmission line of the electrical system.
[0008] The data obtained may have the same structure as the data training data (set). The data obtained may have the same features and / or labels as the training data (set). In other words, for each time series in the at least subset of the training data set, the data obtained may (also) include multiple time series, each of the multiple time series includes multiple labeled time samples, each of the multiple labeled time samples includes the value of each of one or more features at a specific time within the time frame of the time series, and a label indicating that the time sample has been classified into a category in multiple categories. However, the technician should understand that these time series are not the same as the time series of the training data set. In particular, the proportion of representative categories of the data training data (set) and the obtained data may be different. For example, the observed data set will be unbalanced, and the training data set is balanced by the above method. The data obtained may include multiple time series or one or more time series. The data obtained may involve the same electrical system or the same category of electrical system as the training data.
[0009] The method may further include: after adding the generated synthetic time series to the training dataset, and before training the machine learning algorithm, removing one or more majority time series from the training dataset, wherein a majority of the plurality of labeled time samples include labels that do not represent a minority class.
[0010] Removal can be performed until a maximum number of iterations is reached.
[0011] The removal may be performed until a ratio between the number of minority time series in the training dataset and the number of majority time series in the training dataset is within a tolerance of a threshold, wherein a minority time series is defined as a time series in which a majority of the plurality of labeled time samples include a label representing a minority class.
[0012] Receiving at least a subset of the training data set may include selecting a random sample including a predefined number of time series from the training data set.
[0013] Receiving at least a subset of the training data set may include: selecting samples including a predefined number of time series from the training data set to include a higher proportion of minority time series than non-minority time series relative to the training data set, wherein minority time series are defined as time series in which a majority of the multiple labeled time samples include labels representing minority classes, and non-minority time series are defined as time series in which a majority of the multiple labeled time samples include labels representing one of the multiple classes that is not a minority class.
[0014] The plurality of categories may include two categories.
[0015] The number of adjacent time series may be a predefined number greater than one.
[0016] The method may further include: using the at least one hardware processor to generate a predefined number of synthetic time series for each time series in the at least subset of the training data set. Generating the predefined number of synthetic time series may include: randomly selecting one of the identified number of adjacent time series for generating the synthetic time series until the predefined number of synthetic time series has been generated.
[0017] For each of the one or more features and for each of the multiple labeled time samples in the adjacent time series, generating a composite value of the feature may include: when the label of the corresponding labeled time sample in the time series represents a minority class among the multiple classes, generating the composite value of the feature according to a first bias toward the minority class, and, when the label of the corresponding labeled time sample in the time series does not represent the minority class, generating the composite value of the feature according to a second bias that is less biased toward the minority class than the first bias. The minority class may be defined as a class among the multiple classes that is less represented by labels in the multiple time samples in the multiple time series in the training data set than at least one other class among the multiple classes. Generating the composite value of the feature according to the first bias may include: adding a randomly generated value to the value of the feature in the corresponding labeled time sample in the time series. Generating a composite value of the feature according to the second bias may include: calculating the distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series; randomly generating a scaling factor; calculating the scaled value by multiplying the calculated distance by the randomly generated scaling factor; and adding the scaled value to the value of the feature in the corresponding labeled time sample in the time series.
[0018] For each of the one or more features and for each of the multiple labeled time samples in the adjacent time series, generating a composite value of the feature may include: when the label of the corresponding labeled time sample in the time series represents a minority class among the multiple classes, generating the composite value of the feature based on the value of the feature in the corresponding labeled time sample in the time series, and, when the label of the corresponding labeled time sample in the time series does not represent a minority class, generating the composite value of the feature based on both the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series. The minority class may be defined as a class among the multiple classes, that is, the class is less represented by the labels in the multiple time samples in the multiple time series in the training data set than at least one other class among the multiple classes. For example, the minority class may be defined as a class among the multiple classes, that is, the class is least represented by the labels in the multiple time samples in the multiple time series in the training data set. Generating a composite value of the feature based on the value of the feature in the corresponding labeled time sample in the time series may include: adding a randomly generated value to the value of the feature in the corresponding labeled time sample in the time series. Generating a composite value of the feature based on both the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series may include: calculating the distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series; randomly generating a scaling factor; calculating the scaled value by multiplying the calculated distance by the randomly generated scaling factor; and adding the scaled value to the value of the feature in the corresponding labeled time sample in the time series.
[0019] When the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series represents the minority class, the range of the scaling value may be limited to produce a scaled value that is closer to the value of the feature in the labeled time sample in the adjacent time series than to the value of the feature in the corresponding labeled time sample in the time series. When the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series does not represent the minority class, the range of the scaling value may be limited to produce a scaled value that is closer to the value of the feature in the corresponding labeled time sample in the time series than to the value of the feature in the labeled time sample in the adjacent time series.
[0020] Generating a synthetic time series may include: marking the synthetic time series with a label value, the label value being derived based on a label indicating that the time series is classified into one of the multiple categories and a label indicating that an adjacent time series is classified into one of the multiple categories. The label value may be further derived based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and the adjacent time series. When the multiple categories consist of two categories, the label value may be derived by binarizing a value based on the distance between the synthetic time series and the time series and the distance between the synthetic time series and the adjacent time series.
[0021] The method may further include using the at least one hardware processor to train a machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset.
[0022] The one or more features may represent parameters of the electrical system, wherein at least one of the plurality of categories represents an action to be taken in response to an anomaly in the electrical system.
[0023] The method may further include using the at least one hardware processor to randomly select the one or more neighboring time series from the identified number of neighboring time series for each time series in the at least subset of the training data set.
[0024] The method may further include: using the at least one hardware processor to remove one or more majority time series (wherein the majority of the multiple labeled time samples include labels that do not represent the minority class) from the training dataset through one or more iterations until the ratio between the number of minority time series in the training dataset (wherein the majority of the multiple labeled time samples include labels that represent the minority class) and the number of majority time series in the training dataset (wherein the majority of the multiple labeled time samples include labels that represent the majority class) is within a tolerance of a threshold or a maximum number of iterations is reached.
[0025] The distance metric may compare the feature vectors of the plurality of time samples in the two time series to calculate a non-negative scalar value representing the distance between the two time series.
[0026] The method may further include: using the at least one hardware processor to perform the process through one or more iterations until the ratio between the number of minority time series in the training dataset (where the majority of the multiple labeled time samples include labels representing the minority class) and the number of majority time series in the training dataset (where the majority of the multiple labeled time samples include labels representing the majority class) is within a tolerance of a threshold.
[0027] The following list of aspects provides alternative and / or further features of the present disclosure:
[0028] 1. A method for reducing class imbalance in a training dataset for machine learning, the method comprising performing a process using at least one hardware processor to:
[0029] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories;
[0030] For each time series in the at least subset of the training dataset,
[0031] identifying a number of adjacent time series in the at least a subset of the training data set based on a distance metric,
[0032] For each of one or more of the adjacent time series,
[0033] for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and
[0034] generating a synthetic time series from the synthetic time samples including synthetic values of the one or more features; and
[0035] Add the generated synthetic time series to the training dataset.
[0036] 2. The method of aspect 1, wherein receiving at least a subset of the training data set comprises: selecting a random sample comprising a predefined number of time series from the training data set.
[0037] 3. A method according to aspect 1, wherein receiving at least a subset of the training data set may include: selecting samples including a predefined number of time series from the training data set to include a higher proportion of minority time series than non-minority time series relative to the training data set, wherein minority time series are defined as time series in which a majority of the multiple labeled time samples include labels representing minority classes, and non-minority time series are defined as time series in which a majority of the multiple labeled time samples include labels representing one of the multiple classes that is not a minority class.
[0038] 4. The method according to aspect 1, wherein the plurality of categories includes two categories.
[0039] 5. The method according to aspect 1, wherein the number of adjacent time series is a predefined number greater than one.
[0040] 6. The method of aspect 1, further comprising: using the at least one hardware processor to generate a predefined number of synthetic time series for each time series in the at least subset of the training data set.
[0041] 7. The method of clause 6, wherein generating a predefined number of synthetic time sequences comprises randomly selecting one of the identified number of adjacent time sequences for generating the synthetic time sequence until the predefined number of synthetic time sequences have been generated.
[0042] 8. The method according to aspect 1, wherein for each of the plurality of marked time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature comprises:
[0043] When the label of the corresponding labeled time sample in the time series represents a minority class among the plurality of classes, generating a synthetic value of the feature according to a first bias toward the minority class, and
[0044] When the label of the corresponding labeled time sample in the time series does not represent the minority class, a synthetic value of the feature is generated according to a second bias that is less biased toward the minority class than the first bias.
[0045] 9. The method according to aspect 8, wherein the minority class is defined as a class among the multiple classes that is less represented by labels in the multiple time samples in the multiple time series in the training dataset than at least one other class among the multiple classes.
[0046] 10. The method according to aspect 9, wherein generating a synthetic value of the feature according to the first bias comprises: adding a randomly generated value to a value of the feature in a corresponding labeled time sample in the time series.
[0047] 11. The method according to aspect 9, wherein generating a composite value of the feature according to the second bias comprises:
[0048] Calculate the distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series;
[0049] Randomly generate scaling factors;
[0050] calculating a scaled value by multiplying the calculated distance by a randomly generated scaling factor; and
[0051] The scaled value is added to the value of the feature in the corresponding labeled time sample in the time series.
[0052] 12. A method according to aspect 11, wherein, when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series represents the minority class, the range of the scaling value is limited to produce a scaled value that is closer to the value of the feature in the labeled time sample in the adjacent time series than to the value of the feature in the corresponding labeled time sample in the time series.
[0053] 13. A method according to aspect 11, wherein, when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series does not represent the minority class, the range of the scaling value is limited to produce a scaled value that is closer to the value of the feature in the corresponding labeled time sample in the time series than to the value of the feature in the labeled time sample in the adjacent time series.
[0054] 14. A method according to aspect 1, wherein generating a synthetic time series includes: marking the synthetic time series with a label value, wherein the label value is derived based on a label indicating that the time series is classified into one of the multiple categories, and a label indicating that an adjacent time series is classified into one of the multiple categories.
[0055] 15. The method according to clause 14, wherein the label value is further derived based on the distance between the synthetic time series and the time series and the distance between the synthetic time series and the neighboring time series.
[0056] 16. The method according to clause 15, wherein the plurality of categories include two categories, and the label value is derived by binarizing a value based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and an adjacent time series.
[0057] 17. The method of aspect 1, further comprising: using the at least one hardware processor to train a machine learning algorithm using at least a portion of the training dataset after adding the generated synthetic time series to the training dataset.
[0058] 18. The method of aspect 1, wherein the one or more features represent parameters of an electrical system, and wherein at least one of the plurality of categories represents an action to be taken in response to an anomaly in the electrical system.
[0059] 19. A system comprising:
[0060] at least one hardware processor;
[0061] Memory for storing software; and
[0062] software, which is configured to, when executed by the at least one hardware processor, perform the following operations:
[0063] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories,
[0064] For each time series in the at least subset of the training dataset,
[0065] identifying a number of adjacent time series in the at least a subset of the training data set based on a distance metric,
[0066] For each of one or more of the adjacent time series,
[0067] for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a synthetic value of the feature in a synthetic time sample, and generating a synthetic time series from the synthetic time samples including the synthetic values of the one or more features, and
[0068] Add the generated synthetic time series to the training dataset.
[0069] 20. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to:
[0070] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories;
[0071] For each time series in the at least subset of the training dataset,
[0072] identifying a number of adjacent time series in the at least a subset of the training data set based on a distance metric,
[0073] For each of one or more of the adjacent time series,
[0074] for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and
[0075] generating a synthetic time series from the synthetic time samples including synthetic values of the one or more features; and
[0076] Add the generated synthetic time series to the training dataset.
[0077] 21. A computer-implemented method for detecting an event in an electrical system, the method comprising executing, using at least one hardware processor, a process to:
[0078] Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps:
[0079] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories,
[0080] For each time series in the at least subset of the training dataset,
[0081] identifying a number of adjacent time series in the at least a subset of the training data set based on a distance metric,
[0082] For each of one or more of the adjacent time series,
[0083] for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a synthetic value of the feature in a synthetic time sample, and generating a synthetic time series from the synthetic time samples including the synthetic values of the one or more features, and
[0084] Add the generated synthetic time series to the training dataset;
[0085] After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset;
[0086] executing a trained machine learning algorithm on the acquired data related to the electrical system; and
[0087] The event is detected based on execution of a trained machine learning algorithm.
[0088] 22. The method according to aspect 21, wherein the event is an anomaly in the electrical system, in particular a fault in at least one transmission line of the electrical system.
[0089] 23. The method according to aspect 21 or 22, further comprising:
[0090] After adding the generated synthetic time series to the training dataset, and before training the machine learning algorithm, one or more majority time series are removed from the training dataset, wherein a majority of the plurality of labeled time samples include labels that are not representative of the minority class.
[0091] 24. A method according to clause 23, wherein the removing is performed until a maximum number of iterations is reached.
[0092] 25. A method according to aspect 23, wherein the removal is performed until the ratio between the number of minority time series in the training dataset and the number of majority time series in the training dataset is within a tolerance of a threshold, wherein the minority time series is defined as a time series in which a majority of the plurality of labeled time samples include a label representing a minority class.
[0093] 26. A method according to any one of aspects 21 to 25, wherein receiving at least a subset of the training data set comprises: selecting a random sample including a predefined number of time series from the training data set.
[0094] 27. A method according to any one of aspects 21 to 26, wherein receiving at least a subset of the training data set includes: selecting samples including a predefined number of time series from the training data set to include a higher proportion of minority time series than majority time series relative to the training data set.
[0095] 28. A method according to any one of aspects 21 to 27, wherein the plurality of categories comprises two categories.
[0096] 29. A method according to any one of aspects 21 to 28, wherein the number of adjacent time series is a predefined number greater than one.
[0097] 30. The method of any one of clauses 21 to 29, further comprising: using the at least one hardware processor to generate a predefined number of synthetic time series for each time series in the at least subset of the training data set.
[0098] 31. A method according to clause 30, wherein generating a predefined number of synthetic time sequences comprises randomly selecting one of the identified number of adjacent time sequences for generating the synthetic time sequence until the predefined number of synthetic time sequences have been generated.
[0099] 32. The method according to any one of aspects 21 to 31, wherein for each of the plurality of marked time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature comprises:
[0100] When the label of the corresponding labeled time sample in the time series represents a minority class among the plurality of classes, generating a synthetic value of the feature according to a first bias toward the minority class, and
[0101] When the label of the corresponding labeled time sample in the time series does not represent the minority class, a synthetic value of the feature is generated according to a second bias that is less biased toward the minority class than the first bias.
[0102] 33. A method according to aspect 32, wherein generating a composite value of the feature according to the first bias comprises: adding a randomly generated value to a value of the feature in a corresponding labeled time sample in the time series.
[0103] 34. The method according to clause 32, wherein generating a composite value of the feature according to the second bias comprises:
[0104] Calculate the distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in the adjacent time series;
[0105] Randomly generate scaling factors;
[0106] calculating a scaled value by multiplying the calculated distance by a randomly generated scaling factor; and
[0107] The scaled value is added to the value of the feature in the corresponding labeled time sample in the time series.
[0108] 35. A method according to aspect 34, wherein, when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series represents the minority class, the range of the scaling value is limited to produce a scaled value that is closer to the value of the feature in the labeled time sample in the adjacent time series than to the value of the feature in the corresponding labeled time sample in the time series.
[0109] 36. A method according to aspect 34, wherein, when the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series does not represent the minority class, the range of the scaling value is limited to produce a scaled value that is closer to the value of the feature in the corresponding labeled time sample in the time series than to the value of the feature in the labeled time sample in the adjacent time series.
[0110] 37. A method according to any one of aspects 21 to 36, wherein generating a synthetic time series includes: marking the synthetic time series with a label value, which is derived based on a label indicating that the time series is classified into one of the multiple categories, and a label indicating that an adjacent time series is classified into one of the multiple categories.
[0111] 38. A method according to clause 37, wherein the label value is derived further based on the distance between the synthetic time series and the time series and the distance between the synthetic time series and the adjacent time series.
[0112] 39. A method according to clause 38, wherein the plurality of categories comprises two categories, and the label value is derived by binarizing a value based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and an adjacent time series.
[0113] 40. A method according to any one of aspects 1 to 39, wherein the one or more features represent parameters of an electrical system, and wherein at least one of the plurality of categories represents an action to be taken in response to an anomaly in the electrical system.
[0114] 41. A system comprising:
[0115] at least one hardware processor;
[0116] Memory for storing software; and
[0117] software, which is configured to, when executed by the at least one hardware processor, perform the following operations:
[0118] Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps:
[0119] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories,
[0120] For each time series in the at least subset of the training dataset,
[0121] identifying a number of neighboring time series in the at least a subset of the training data set based on a distance metric, for each of one or more of the neighboring time series,
[0122] for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a synthetic value of the feature in a synthetic time sample, and generating a synthetic time series from the synthetic time samples including the synthetic values of the one or more features, and
[0123] Add the generated synthetic time series to the training dataset;
[0124] After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset;
[0125] executing a trained machine learning algorithm on the acquired data related to the electrical system; and
[0126] The event is detected based on execution of a trained machine learning algorithm.
[0127] 42. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to:
[0128] Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps:
[0129] receiving at least a subset of a training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories,
[0130] For each time series in the at least subset of the training dataset,
[0131] identifying a number of adjacent time series in the at least a subset of the training data set based on a distance metric,
[0132] For each of one or more of the adjacent time series,
[0133] For each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and generating a composite time series from the composite time samples including the composite values of the one or more features; and
[0134] Add the generated synthetic time series to the training dataset;
[0135] After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset;
[0136] executing a trained machine learning algorithm on the acquired data related to the electrical system; and
[0137] The event is detected based on execution of a trained machine learning algorithm.
[0138] Any of the above methods may be embodied, alone or in any combination, in an executable software module of a processor-based system (such as a server) and / or in executable instructions stored in a non-transitory computer-readable medium. BRIEF DESCRIPTION OF THE DRAWINGS
[0139] The details of the present disclosure, both as to its structure and operation, may be gleaned in part by examination of the accompanying drawings, wherein like reference numerals refer to like parts, and wherein:
[0140] Figure 1 illustrates an example infrastructure in which one or more of the processes described herein may be implemented according to an embodiment;
[0141] Figure 2 An example processing system is illustrated according to an embodiment by which one or more of the processes described herein may be performed;
[0142] Figure 3 A resampling process for reducing class imbalance in a training dataset for machine learning using upsampling according to an embodiment is illustrated;
[0143] Figure 4 A resampling process for reducing class imbalance in a training dataset for machine learning using downsampling according to an embodiment is illustrated;
[0144] Figure 5 illustrates a resampling process for reducing class imbalance in a training dataset for machine learning using both upsampling and downsampling according to an embodiment; and
[0145] Figure 6 Illustrated is a comparison between precision-recall curves obtained for machine learning models trained without a resampled dataset and with a dataset resampled according to an embodiment of the disclosed resampling process. DETAILED DESCRIPTION
[0146] In an embodiment, a system, method, and non-transitory computer-readable medium for reducing class imbalance in a training data set for machine learning are disclosed. After reading this description, it will become apparent to those skilled in the art how to implement the present disclosure in various alternative embodiments and alternative applications. However, although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are presented only by way of example and illustration, not limitation. Thus, this detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure as set forth in the appended claims.
[0147] 1. System Overview
[0148] In an embodiment, a system, method, and non-transitory computer-readable medium for reducing class imbalance in a training data set for machine learning are disclosed. After reading this description, it will become apparent to those skilled in the art how to implement the present disclosure in various alternative embodiments and alternative applications. However, although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are presented only by way of example and illustration, not limitation. Thus, this detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure as set forth in the appended claims.
[0149] 1.1. Infrastructure
[0150] Figure 1 An example infrastructure is illustrated in which one or more of the disclosed processes may be implemented according to an embodiment. The infrastructure may include a platform 110 (e.g., one or more servers) that hosts and / or executes one or more of the various functions, processes, methods, and / or software modules described herein. The platform 110 may include a dedicated server, or may instead include a cloud instance that utilizes shared resources of one or more servers. These servers or cloud instances may be collocated and / or geographically distributed. The platform 110 may also include or be communicatively connected to a server application 112 and / or one or more databases 114. In addition, the platform 110 may be communicatively connected to one or more user systems 130 via one or more networks 120. The platform 110 may also be communicatively connected to one or more external systems 140 (e.g., other platforms, websites, etc.) via one or more networks 120.
[0151] The network(s) 120 may include the Internet, and the platform 110 may communicate with the user system(s) 130 over the Internet using standard transfer protocols such as Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), File Transfer Protocol (FTP), FTP Secure (FTPS), Secure Shell FTP (SFTP), etc., as well as proprietary protocols. Although the platform 110 is illustrated as being connected to various systems via a single set of network(s) 120, it should be understood that the platform 110 may be connected to various systems via a different set of one or more networks. For example, the platform 110 may be connected to a subset of the user systems 130 and / or external systems 140 via the Internet, but may be connected to one or more other user systems 130 and / or external systems 140 via an intranet. Furthermore, although only a few user systems 130 and external systems 140, one server application 112, and a set of database(s) 114 are illustrated, it should be understood that the infrastructure may include any number of user systems, external systems, server applications, and databases.
[0152] The user system(s) 130 may include any one or more types of computing devices capable of wired and / or wireless communication, including, but not limited to, desktop computers, laptop computers, tablet computers, smartphones or other mobile phones, servers, game consoles, televisions, set-top boxes, electronic kiosks, point-of-sale terminals, and / or the like. Each user system 130 may include or be communicatively connected to a client application 132 and / or one or more local databases 134. In general, it is contemplated that the user system(s) 130 will include personal computers or professional workstations that users may utilize to connect to and interact with the platform 110 to resample training data sets that are stored on the platform 110 (e.g., in the database(s) 114) and / or used to train machine learning models hosted on or deployed from the platform 110, as disclosed herein.
[0153] The platform 110 may include a web server hosting one or more websites and / or web services. In embodiments where a website is provided, the website may include a graphical user interface, which includes, for example, one or more screens (e.g., web pages) generated in a hypertext markup language (HTML) or other language. The platform 110 transmits or serves one or more screens of a graphical user interface in response to a request from (multiple) user systems 130. In some embodiments, these screens may be provided in the form of a wizard, in which case two or more screens may be provided in a sequential manner, and one or more of the sequential screens may depend on the interaction of the user or user system 130 with one or more previous screens. Both requests to the platform 110 and responses from the platform 110 (including screens of a graphical user interface) may be communicated through (multiple) networks 120 (which may include the Internet) using standard communication protocols (e.g., HTTP, HTTPS, etc.). These screens (e.g., web pages) may include a combination of content and elements such as text, images, videos, animations, references (e.g., hyperlinks), frames, inputs (e.g., text boxes, text areas, check boxes, radio buttons, drop-down menus, buttons, forms, etc.), scripts (e.g., JavaScript), etc., including elements that include or are derived from data stored in one or more databases (e.g., database(s) 114) that may be accessed locally and / or remotely by the platform 110. The platform 110 may also respond to other requests from the user system(s) 130.
[0154] The platform 110 may include, be communicatively coupled to, or otherwise access one or more databases 114. For example, the platform 110 may include one or more database servers that manage the one or more databases 114. A server application 112 executing on the platform 110 and / or a client application 132 executing on a user system 130 may submit data (e.g., user data, form data, etc.) to be stored in the (multiple) databases 114, and / or request access to data stored in the (multiple) databases 114. Any suitable database may be utilized, including, but not limited to, MySQL. TM 、Oracle TM 、IBM TM , Microsoft SQL TM 、Access TM , PostgreSQL TMThe data may be sent to the platform 110 using, for example, a well-known POST request supported by HTTP, via FTP, and / or the like. The data and other requests may be processed, for example, by server-side network technology such as a servlet or other software module executed by the platform 110 (e.g., included in the server application 112).
[0155] In embodiments where a network service is provided, the platform 110 may receive requests from (multiple) external systems 140 and provide responses in extensible markup language (XML), JavaScript object notation (JSON), and / or any other suitable or desired format. In such embodiments, the platform 110 may provide an application programming interface (API) that defines how (multiple) user systems 130 and / or (multiple) external systems 140 interact with the network service. Thus, (multiple) user systems 130 and / or (multiple) external systems 140 (which may themselves be servers) may define their own user interfaces and rely on the network service to implement or otherwise provide the backend processes, methods, functionality, storage, and / or the like described herein. For example, in such embodiments, a client application 132 executing on one or more user systems 130 may interact with a server application 112 executing on the platform 110 to perform one or more or one or more of the various functions, processes, methods, and / or software modules described herein. In an embodiment, the client application 132 may utilize a local database 134 to store data locally on the user system 130.
[0156] The client applications 132 may be "thin", in which case the processing is primarily performed on the server side by the server applications 112 on the platform 110. A basic example of a thin client application 132 is a browser application that simply requests, receives, and renders web pages on the user system(s) 130, while the server applications 112 on the platform 110 are responsible for generating the web pages and managing database functions. Alternatively, the client application may be "thick", in which case the processing is primarily performed on the client side by the user system(s) 130. It should be understood that the client application 132 may perform some amount of processing relative to the server application 112 on the platform 110 at any point along this spectrum between "thin" and "thick", depending on the design goals of the particular implementation. In any case, the software described herein may reside entirely on the platform 110 (e.g., in which case the server application 112 performs all processing) or on the (multiple) user system 130 (e.g., in which case the client application 132 performs all processing), or distributed between the platform 110 and the (multiple) user system 130 (e.g., in which case both the server application 112 and the client application 132 perform processing), and may include one or more executable software modules that include instructions for implementing one or more of the processes, methods, or functions described herein.
[0157] In an embodiment, a system, method, and non-transitory computer-readable medium for reducing class imbalance in a training data set for machine learning are disclosed. After reading this description, it will become apparent to those skilled in the art how to implement the present disclosure in various alternative embodiments and alternative applications. However, although various embodiments of the present disclosure will be described herein, it should be understood that these embodiments are presented only by way of example and illustration, not limitation. Thus, this detailed description of various embodiments should not be interpreted as limiting the scope or breadth of the present disclosure as set forth in the appended claims.
[0158] 1.2. Example Processing Device
[0159] Figure 2is a block diagram illustrating an example wired or wireless system 200 that may be used in conjunction with the various embodiments described herein. For example, the system 200 may be used as or in conjunction with one or more of the functions, processes, or methods (e.g., to store and / or execute software) described herein, and may represent a component of the platform 110, (multiple) user system 130, (multiple) external system 140, and / or other processing devices described herein. The system 200 may be a server or any conventional personal computer, or any other processor-enabled device capable of wired or wireless data communications. Other computer systems and / or architectures may also be used, as will be apparent to those skilled in the art.
[0160] The system 200 preferably includes one or more processors 210. The processor(s) 210 may include a central processing unit (CPU). Additional processors may be provided, such as a graphics processing unit (GPU), an auxiliary processor to manage input / output, an auxiliary processor to perform floating point math operations, a dedicated microprocessor with an architecture suitable for fast execution of signal processing algorithms (e.g., a digital signal processor), a slave processor subordinate to a main processing system (e.g., a back end processor), an additional microprocessor or controller for a dual processor or multi-processor system, and / or a coprocessor. Such auxiliary processors may be discrete processors or may be integrated with the processor 210. Examples of processors that may be used with the system 200 include, but are not limited to, any processor available from Intel Corporation of Santa Clara, California (e.g., Pentium TM 、Core i7 TM , Xeon TM any processor available from Advanced Micro Devices, Inc. (AMD) of Santa Clara, California, any processor available from Apple Inc. of Cupertino (e.g., A-series, M-series, etc.), any processor available from Samsung Electronics Co., Ltd. of Seoul, South Korea (e.g., Exynos TM ), any processor available from NXP Semiconductors NV of Eindhoven, the Netherlands, and / or the like.
[0161] The processor 210 is preferably connected to a communication bus 205. The communication bus 205 may include a data channel for facilitating information transfer between storage devices and other peripheral components of the system 200. In addition, the communication bus 205 may provide a set of signals for communicating with the processor 210, including a data bus, an address bus, and / or a control bus (not shown). The communication bus 205 may include any standard or non-standard bus architecture, such as, for example, a bus architecture that conforms to the following: Industry Standard Architecture (ISA), Extended Industry Standard Architecture (EISA), Micro Channel Architecture (MCA), Peripheral Component Interconnect (PCI) local bus, standards published by the Institute of Electrical and Electronics Engineers (IEEE) (including IEEE 488 General Purpose Interface Bus (GPIB), IEEE 696 / S-100, and / or the like).
[0162] The system 200 preferably includes a main memory 215 and may also include a secondary memory 220. The main memory 215 provides storage of instructions and data for programs executed on the processor 210 (such as any software discussed herein). It should be understood that the programs stored in the memory and executed by the processor 210 can be written and / or compiled according to any suitable language (including but not limited to, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, etc.). The main memory 215 is typically a semiconductor-based memory, such as a dynamic random access memory (DRAM) and / or a static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), etc., including read-only memory (ROM).
[0163] The secondary memory 220 is a non-transitory computer-readable medium having computer executable code (e.g., any software disclosed herein) and / or other data stored thereon. The computer software or data stored on the secondary memory 220 is read into the main memory 215 for execution by the processor 210. The secondary memory 220 may include, for example, semiconductor-based memories such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), and flash memory (block-oriented memory similar to EEPROM).
[0164] The secondary storage 220 may optionally include internal media 225 and / or removable media 230. Reading from and / or writing to the removable media 230 is performed in any known manner. The removable storage medium 230 may be, for example, a tape drive, a compact disk (CD) drive, a digital versatile disk (DVD) drive, other optical drives, a flash memory drive, and / or the like.
[0165] In alternative embodiments, the secondary memory 220 may include other similar devices for allowing computer programs or other data or instructions to be loaded into the system 200. Such devices may include, for example, a communication interface 240 that allows software and data to be transferred from an external storage medium 245 to the system 200. Examples of external storage media 245 include an external hard drive, an external optical drive, an external magneto-optical drive, and / or the like.
[0166] As mentioned above, the system 200 may include a communication interface 240. The communication interface 240 allows software and data to be transferred between the system 200 and an external device (e.g., a printer), a network, or other information source. For example, computer software or executable code may be transferred from a network server (e.g., platform 110) to the system 200 via the communication interface 240. Examples of the communication interface 240 include a built-in network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a cardbus network adapter, a wireless network adapter, a universal serial bus (USB) network adapter, a modem, a wireless data card, a communication port, an infrared interface, an IEEE 1394 FireWire, and any other device capable of connecting the system 200 to a network (e.g., network(s) 120) or another computing device. The communication interface 240 preferably implements industry-published protocol standards, such as Ethernet IEEE 802 standards, Fibre Channel, Digital Subscriber Line (DSL), Asynchronous Digital Subscriber Line (ADSL), Frame Relay, Asynchronous Transfer Mode (ATM), Integrated Services Digital Network (ISDN), Personal Communications Service (PCS), Transmission Control Protocol / Internet Protocol (TCP / IP), Serial Line Internet Protocol / Point-to-Point Protocol (SLIP / PPP), etc., but may also implement custom or non-standard interface protocols.
[0167] Software and data transmitted via communication interface 240 are typically in the form of electrical communication signals 255. These signals 255 may be provided to communication interface 240 via communication channel 250. In embodiments, communication channel 250 may be a wired or wireless network (e.g., network(s) 120) or any other communication link of any kind. Communication channel 250 carries signals 255 and may be implemented using a variety of wired or wireless communication means, including wire or cable, fiber optics, a conventional telephone line, a cellular telephone link, a wireless data communication link, a radio frequency ("RF") link, or an infrared link, to name a few.
[0168] Computer executable code (e.g., computer programs, such as the disclosed software) is stored in the main memory 215 and / or the secondary memory 220. Computer executable code may also be received via the communication interface 240 and stored in the main memory 215 and / or the secondary memory 220. When executed, such computer programs enable the system 200 to perform various functions of the disclosed embodiments as described elsewhere herein.
[0169] In this description, the term "computer-readable medium" is used to refer to any non-transitory computer-readable storage medium used to provide computer executable code and / or other data to or within the system 200. Examples of such media include main memory 215, secondary memory 220 (including internal memory 225, removable media 230, and external storage media 245), and any peripheral devices (including network information servers or other network devices) communicatively coupled to communication interface 240. These non-transitory computer-readable media are means for providing software and / or other data to system 200.
[0170] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into the system 200 via the removable medium 230, the I / O interface 235, or the communication interface 240. In such embodiments, the software is loaded into the system 200 in the form of electrical communication signals 255. When executed by the processor 210, the software preferably causes the processor 210 to perform one or more of the processes and functions described elsewhere herein.
[0171] In an embodiment, the I / O interface 235 provides an interface between one or more components of the system 200 and one or more input and / or output devices. Example input devices include, but are not limited to, sensors, keyboards, touch screens or other touch-sensitive devices, cameras, biometric sensing devices, computer mice, trackballs, pen-based pointing devices, and / or the like. Examples of output devices include, but are not limited to, other processing devices, cathode ray tubes (CRTs), plasma displays, light emitting diode (LED) displays, liquid crystal displays (LCDs), printers, vacuum fluorescent displays (VFDs), surface conduction electron emitter displays (SEDs), field emission displays (FEDs), and / or the like. In some cases, input and output devices may be combined, such as in the case of a touch panel display (e.g., in a smartphone, tablet, or other mobile device).
[0172] System 200 may also include optional wireless communication components that facilitate wireless communication over a voice network and / or a data network (e.g., in the case of user system 130). The wireless communication components include antenna system 270, radio system 265, and baseband system 260. In system 200, radio frequency (RF) signals are transmitted and received over the air by antenna system 270 under the management of radio system 265.
[0173] In an embodiment, antenna system 270 may include one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmit and receive signal paths for antenna system 270. In the receive path, a received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown) that amplifies the received RF signal and sends the amplified signal to radio system 265.
[0174] In an alternative embodiment, the radio system 265 may include one or more radios configured to communicate over various frequencies. In an embodiment, the radio system 265 may combine a demodulator (not shown) and a modulator (not shown) in one integrated circuit (IC). The demodulator and modulator may also be separate components. In the incoming path, the demodulator strips off the RF carrier signal, leaving a baseband receive audio signal that is sent from the radio system 265 to the baseband system 260.
[0175] If the received signal includes audio information, the baseband system 260 decodes the signal and converts it into an analog signal. The signal is then amplified and sent to a speaker. The baseband system 260 also receives analog audio signals from a microphone. These analog audio signals are converted into digital signals and encoded by the baseband system 260. The baseband system 260 also encodes the digital signal for transmission and generates a baseband transmit audio signal that is routed to the modulator portion of the radio system 265. The modulator mixes the baseband transmit audio signal with an RF carrier signal to generate an RF transmit signal that is routed to the antenna system 270 and can pass through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 270, where the signal is switched to the antenna port for transmission.
[0176] The baseband system 260 is also communicatively coupled to the processor(s) 210. The processor(s) 210 may access data storage areas 215 and 220. The processor(s) 210 are preferably configured to execute instructions (i.e., computer programs, such as the disclosed software) which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. When executed, such computer programs may enable the system 200 to perform various functions of the disclosed embodiments.
[0177] 2. Process Overview
[0178] An embodiment of a process for reducing class imbalance in a training data set for machine learning will now be described in detail. It should be understood that the described process may be embodied in one or more software modules executed by one or more hardware processors (e.g., processor 210), for example as a software application (e.g., server application 112, client application 132, and / or a distributed application including both server application 112 and client application 132), which may be executed entirely by the processor(s) of platform 110, entirely by the processor(s) of user system(s) 130, or may be distributed across platform 110 and user system(s) 130, such that some parts or modules of the software application are executed by platform 110 and other parts or modules of the software application are executed by user system(s) 130. The described process may be implemented as instructions represented in source code, object code, and / or machine code. These instructions may be executed directly by hardware processor(s) 210, or alternatively, may be executed by a virtual machine operating between the object code and hardware processor(s) 210. In addition, the disclosed software may be built on or connected to one or more existing systems.
[0179] Alternatively, the described process may be implemented as a hardware component (e.g., a general purpose processor, an integrated circuit (IC), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, etc.), a combination of hardware components, or a combination of hardware and software components. In order to clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps are generally described in terms of their functionality herein. Whether such functionality is implemented as hardware or software depends on a specific application and the design constraints imposed on the entire system. Skilled personnel may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. In addition, the functional grouping within a component, block, module, circuit or step is for ease of description. Without departing from the present disclosure, a specific function or step may be moved from one component, block, module, circuit or step to another.
[0180] In addition, although the processes described herein are illustrated using certain arrangements and sequencing of subprocesses, each process may be implemented with fewer, more, or different subprocesses and different arrangements and / or sequencing of subprocesses. In addition, it should be understood that any subprocess that is not dependent on the completion of another subprocess may be performed before, after, or in parallel with other independent subprocesses, even if the subprocesses are described or illustrated in a particular order.
[0181] The baseband system 260 is also communicatively coupled to the processor(s) 210. The processor(s) 210 may access data storage areas 215 and 220. The processor(s) 210 are preferably configured to execute instructions (i.e., computer programs, such as the disclosed software) which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. When executed, such computer programs may enable the system 200 to perform various functions of the disclosed embodiments.
[0182] 2.1. introduction
[0183] Before training a machine learning model (such as a classifier), it is important to balance the training dataset. An unbalanced training dataset will bias the resulting model towards the majority class because the model will be primarily under the influence of data representing the majority class. "Resampling" refers to a data preprocessing method that mitigates the imbalance in a training dataset by generating synthetic data that mimics the data in the training dataset. The synthetic data for the minority class modifies the overall distribution of class labels in the training dataset toward a more balanced distribution. An example of a resampling method is the synthetic minority oversampling technique (SMOTE), which is described by Nitesh et al. in "SMOTE: Synthetic Minority Oversampling Technique" (Journal of Artificial Intelligence Research, 16:321-357, 2002, which is hereby incorporated by reference into this article as if fully set forth).
[0184] While resampling has been shown to be effective for independent and identically distributed (IID) data, conventional resampling techniques are often not equally effective for time series data. Time series data includes multiple time samples. Any resampling of time series data should take into account the dependencies of these time samples along the time dimension. In addition, because time series data has a time dimension, time series data is three-dimensional, resulting in a dual class imbalance along the batch or case dimension and the time dimension. In other words, the class imbalance of three-dimensional time series data is more severe than that of two-dimensional IID data. For example, with 50 time samples in each time series, the class imbalance in the time series data is 50 times that in the two-dimensional IID data. Therefore, applying traditional interpolation-based resampling (such as SMOTE) to time series data is too memory-intensive and relatively inefficient.
[0185] Thus, disclosed herein are embodiments of a process for resampling time series data. In particular, the disclosed embodiments reduce imbalances in a training data set for time series data, in which time samples of the time series have been labeled with categories for supervised training of a machine learning classifier. For example, a training data set may include multiple time series, each time series may include multiple time samples, and each time sample may include a value for each of one or more features (e.g., a feature vector of feature values), and a label representing the true category into which the time sample has been classified. In an embodiment, all values of all of the multiple features may be normalized to real values between -1 and 1. As will be appreciated by those skilled in the art, the original data set may be labeled in any known manner (e.g., manually annotated by one or more levels of experts).
[0186] In an embodiment, the resampling process may include an upsampling algorithm that generates synthetic time series data. In an alternative or additional embodiment, the resampling process may include a downsampling algorithm that reduces the distribution of the majority class (s) in the existing time series data. It should be understood that the embodiment may consist of only an upsampling algorithm or only a downsampling algorithm, or may include both an upsampling algorithm and a downsampling algorithm. In each embodiment, the goal may be to achieve a ratio of the number of time series belonging to the minority class to the number of majority time series belonging to the majority class in the training data set, which ratio meets a predefined threshold or is within a tolerance range of a predefined threshold. It should be understood that for a binary machine learning classifier (i.e., there are only two possible classes), the ideal value of this ratio is 1.0.
[0187] The baseband system 260 is also communicatively coupled to the processor(s) 210. The processor(s) 210 may access data storage areas 215 and 220. The processor(s) 210 are preferably configured to execute instructions (i.e., computer programs, such as the disclosed software) which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. When executed, such computer programs may enable the system 200 to perform various functions of the disclosed embodiments.
[0188] 2.2. Upsampling algorithm
[0189] The goal of upsampling is to reduce the severity of class imbalance in a data set by synthesizing artificial data representing a minority class. In the case of time series data, each time sample in a synthetic time series representing a minority class can be generated from information close to the real time series representing the minority class. Table 1 below represents a pseudo code for an upsampling algorithm according to an embodiment:
[0190]
[0191]
[0192] Figure 3 The same upsampling algorithm according to an embodiment is illustrated in a flow chart. The upsampling algorithm 300 may be implemented as a software module hosted and / or executed on the platform 110 or the user system 130 for managing training data for a machine learning model, training a machine learning model, deploying a machine learning model, and / or executing a machine learning model. The upsampling algorithm 300 may represent a module of the entire resampling process or an overarching resampling process.
[0193] The upsampling algorithm 300 may accept one or more parameter values as input. For example, the input to the upsampling algorithm 300 may be an original dataset (e.g., X in Table 1), which represents the original, unbalanced training dataset. The upsampling algorithm 300 may also accept one or more hyperparameters, such as the number of time series to be sampled from the original dataset (e.g., n in Table 1), the number of synthetic time series to be generated from each sampled time series in the original dataset (e.g., n in Table 1), and the number of times series to be generated from each sampled time series in the original dataset (e.g., n in Table 1). ts ) and the number of neighbors to use (e.g., k in Table 1, k>0). It should be understood that these hyperparameters can be selected or calculated based on the degree of class imbalance in the dataset. For example, for greater class imbalance, the number of time series to be sampled, the number of synthetic time series to be generated, and / or the number of neighbors to be used can be increased (e.g., n, n respectively in Table 1). ts The output of the upsampling algorithm 300 may be a synthetic dataset (e.g., X in Table 1) to be combined into the original dataset. S ).
[0194] The upsampling algorithm 300 may also receive or initialize one or more parameters. For example, the upsampling algorithm 300 may identify a minority class in the original data set (e.g., row 01 in Table 1). It should be understood that each time sample in each time series in the original data set may be labeled with one of a plurality of classes. For a binary classifier, there are only two classes, such that one class is a minority class and the other class is a majority class. The minority class may be determined as a class among the plurality of classes for which there is a minimum or minimal number of labeled time samples in the original data set. Additionally, the upsampling algorithm 300 may initialize a data structure for the synthetic data set (e.g., row 02 in Table 1).
[0195] In subprocess 305 (e.g., corresponding to row 03 in Table 1), a subset (e.g., S in Table 1) of the original data set (e.g., X in Table 1) may be sampled. Although all time series in the original data set may be sampled instead of just a smaller subset, this may be computationally too expensive for some systems. Therefore, in an embodiment, only a smaller subset of the original data set is sampled. In this case, the time series may be randomly sampled from the original data set. In an embodiment, the sample is weighted to include a higher proportion of time series belonging to the minority class than the original data set, so that the sample has a skewed probability distribution so as to increase the probability of randomly selecting a time series of the minority class relative to the original data set. In the case of a binary classifier, the time series of the minority class may be defined as a time series in which the time samples of the minority class are more than the time samples of the majority class. A subset may be sampled from the original data set so that it consists of a predefined number (e.g., n in Table 1) of time series.
[0196] Subprocess 310 (e.g., corresponding to row 04 in Table 1) may iterate through each time series in the sampled subset (e.g., S in Table 1) from subprocess 305. If there are still time series to be considered from the sampled subset (i.e., "yes" in subprocess 310), algorithm 300 proceeds to subprocess 315 to consider the next time series TS i Otherwise, if there are no time series to be considered from the sampled subset (ie, “no” in subprocess 310 ), algorithm 300 proceeds to subprocess 375 .
[0197] In subprocess 315 (eg, corresponding to row 05 in Table 1), a sampled subset (eg, S in Table 1) is identified that corresponds to the current time series TS under consideration. i In other words, a predefined number (e.g., k in Table 1) of nearest time series in the sampled subset are identified based on the distance metric. It should be understood that although in most cases the nearest time series TS i It may be preferred, but not required, to identify the nearest time series as the set of neighboring time series. In an alternative embodiment, the set of neighboring time series identified in subprocess 315 may include time series that are further away from the current time series TS than those not included in the set of neighboring time series. i For example, a set of neighboring time series may be selected to include the time series TS i Neighbors within a predefined threshold distance, not necessarily requiring them to be closer to the current time series TS than any neighbor not included in the set of neighboring time series i . Therefore, it should be understood that whenever the term "k nearest" is used herein, alternative embodiments may instead utilize a set of nearby time series (e.g., within a threshold distance), without strictly requiring it to be a set of k nearest time series. More generally, the current time series TS i The neighborhood of can be defined in any way to include any number of time series according to any criteria and / or metric.
[0198] Any metric that can measure the correlation or similarity between two time series can be used as a distance metric. For example, a distance metric can measure the Euclidean distance between the feature vectors of the time samples of the two time series. As another example, the distance metric can utilize the following correlation distance corr between two vectors x1 and x2 (e.g., feature vectors):
[0199]
[0200] corr(x1,x2):=1-|ρ(x1,x2)|
[0201] Typically, in an embodiment, a distance metric accepts two time series (e.g., each time series includes one or more feature vectors) as input and outputs a scalar value representing the distance between the two time series. The distance metric should be symmetric so that the distance from the first time series to the second time series is the same as the distance from the second time series to the first time series, as calculated by the distance metric. In addition, the output value of the distance metric should be non-negative, and the distance between a time series and itself should be zero.
[0202] If different time series may have different lengths (e.g., different numbers of time samples), the distance metric may utilize dynamic time warping (as described, for example, in "Dynamic Time Warping" in "Information Retrieval for Music and Sports" (Springer, Berlin, Heidelberg, doi.org / 10.1007 / 978-3-540-74048-3_4), which is hereby incorporated by reference as if fully set forth herein) to compare time series of different lengths. It should be understood that dynamic time warping is not necessary when all time series have the same length.
[0203] Subprocess 320 (eg, corresponding to row 06 in Table 1) may sample a neighboring time series TS from the k nearest neighbors (eg, K in Table 1) identified in subprocess 315 n , until a predefined number (e.g., number_synthetic_TS in Table 1) of synthetic time series TS have been generated s If you want to get the current time series TS from the i If another synthetic time series is generated (ie, "yes" in subprocess 320), algorithm 300 proceeds to subprocess 325 to generate the next time series TS n Otherwise, if there are no more synthetic time series, the current time series TS under consideration will be sampled. i is generated (ie, “no” in subprocess 320 ), the algorithm 300 returns to subprocess 310 .
[0204] In subprocess 325 (eg, corresponding to row 07 in Table 1), a neighboring time series TS is sampled from the k nearest neighbors (eg, K in Table 1) identified in subprocess 315. n In an embodiment, the adjacent time series TS n is randomly selected from a set of k nearest neighbors. However, it should be understood that the neighboring time series TS may be selected in any different manner. n .
[0205] In subprocess 330 (eg, corresponding to row 08 in Table 1), the adjacent time series TS sampled in subprocess 325 is determined. n Whether it belongs to the minority class (e.g., minority in Table 1). In the case of a binary classifier, if the neighboring time series TS n The time samples of the minority class are more than the time samples of the majority class (for example, the adjacent time series TS n More than half of the time samples in are labeled with minority categories), then the adjacent time series TS n It can belong to the minority class. If the adjacent time series TS n belongs to the minority class (ie, "yes" in subprocess 330), the algorithm 300 proceeds to subprocess 335 to begin creating a synthetic time series TS s (For example, initialized in row 09 in Table 1). Otherwise, if the neighboring time series TS n does not belong to the minority class (i.e., "No" in subprocess 330), the algorithm 300 returns to subprocess 320 to potentially sample a new neighboring time series. In this case, the neighboring time series TS sampled in subprocess 325 n was effectively discarded.
[0206] Subprocess 335 (eg, corresponding to row 10 in Table 1) may iterate through each feature f of the plurality of features (eg, the adjacent time series TS sampled in subprocess 325). n It should be understood that each time series in the original dataset should have the same set of features, so that the current time series TS under consideration i and the current neighboring time series TS n Both should have the same values of the features. As a result, the generated synthetic time series TS s If feature f remains to be considered from the plurality of features (i.e., "yes" in subprocess 335), algorithm 300 proceeds to subprocess 340. Otherwise, if no feature remains to be considered from the plurality of features (i.e., "no" in subprocess 335), algorithm 300 proceeds to subprocess 370.
[0207] Subprocess 340 (eg, corresponding to row 11 in Table 1) may iterate through the neighboring time series TS sampled in subprocess 325 n Alternatively, subprocess 340 may iterate through the current time series TS under consideration iDepending on how the original dataset is constructed, the time series in the original dataset may or may not all have the same number of time samples t. If the time series is TS i and TS n do not have the same number of time samples t, then subprocess 340 may be restricted to less than or equal to the number of samples from the time series TS i or TS n The algorithm 300 then iterates through the minimum number of time samples to be considered (e.g., to prevent illegal memory accesses to data structures representing shorter time series). If time samples t remain to be considered (i.e., "yes" in subprocess 340), the algorithm 300 proceeds to subprocess 345. Otherwise, if no time samples remain to be considered (i.e., "no" in subprocess 340), the algorithm 300 returns to subprocess 335 to determine whether another feature remains to be considered.
[0208] It should be understood that the order of subprocesses 335 and 340 can be easily switched so that the algorithm 300 traverses each feature for each time sample, rather than traversing each time sample for each feature. In either case, subprocesses 335 and 340 jointly traverse the two-dimensional space of features and time. In general, the training data set (e.g., X in Table 1) can be viewed as a three-dimensional structure of time series, time samples, and features. Algorithm 300 processes at least a subset of this three-dimensional structure to generate a synthetic time series.
[0209] In subprocess 345 (eg, corresponding to row 12 in Table 1), the current time series TS under consideration is determined i The current feature f of the current time sample t in (i.e., TS i [f][t]) has been labeled with the minority class. If the current time series TS i has been labeled with the minority class (ie, "yes" in subprocess 345), then algorithm 300 proceeds to subprocess 350. Otherwise, if the current time series TS i Not yet labeled with the minority class (ie, "No" in subprocess 345), algorithm 300 proceeds to subprocess 355. It should be understood that for binary classification, time series that are not labeled with the minority class will necessarily be labeled with the majority class.
[0210] In subprocess 350 (eg, corresponding to row 13 in Table 1), based on the current time series TS under consideration, i Feature-time sample TS i [f][t] value to generate synthetic feature-time sample TS s [f][t] value. In an embodiment of subprocess 350, the synthetic feature-time sample TSs The value of [f][t] is based on the current time series TS i Feature-time sample TS i [f][t] (because it is known to belong to the minority class (i.e., by virtue of subprocess 345)), rather than being generated based on the current neighboring time series TS under consideration n Feature-time sample TS n [f][t] is generated. For example, as shown in Table 1, the synthetic feature-time sample TS s The value of [f][t] can be calculated as the current time series TS i Feature-time sample TS i The value of [f][t] is modified based on a random value from a range of possible values. For example, a random value can be added to the current time series TS i Feature-time sample TS i The value of [f][t].
[0211] In Table 1, the selection of random values in subprocess 350 is represented by the rdm() function. The rdm() function can return a random number within the range specified by the input argument, where the random number has a uniform distribution over many iterations (i.e., it is not biased towards any value within the specified range). For example, rdm(0,0.1) returns a uniform random real value between 0 and 0.1. It is worth noting that in subprocess 350, the range of random values is set to be low so that the synthetic feature-time sample TS s The result value of [f][t] will be close to the current time series TS i Feature-time sample TS i [f][t], the current time series belongs to the minority class, as determined in subprocess 345. It should be understood that the range of random values 0.0 to 0.1 is only one example, and other ranges (e.g., 0.0 to 0.2, 0.1 to 0.3, etc.) can be used. In general, the range should be, but need not be, a subset of the values between 0.0 and 0.5, so that the random values are guaranteed to be closer to 0.0 than to 1.0.
[0212] In subprocess 355 (eg, corresponding to row 14 in Table 1), the current neighboring time series TS under consideration is determined. n The current feature f of the current time sample t in (i.e., TS n [f][t]) has been labeled with the minority class. If the current neighboring time series TS n has been labeled with the minority class (ie, "yes" in subprocess 355), then algorithm 300 proceeds to subprocess 360. Otherwise, if the current neighboring time series TS nNot yet labeled with the minority class (ie, “No” in subprocess 355 , and corresponding to row 16 in Table 1), algorithm 300 proceeds to subprocess 365 .
[0213] In subprocess 360 (eg, corresponding to row 15 in Table 1), based on the current time series TS under consideration, i Feature-time sample TS i The value of [f][t] and the current neighboring time series TS under consideration n Feature-time sample TS n The values of [f][t] are used to generate synthetic features-time samples TS s [f][t] value. For example, as shown in Table 1, the synthetic feature-time sample TS s The value of [f][t] can be calculated as the current time series TS i Feature-time sample TS i The value of [f][t] is added to the product of a random value from a set of possible values multiplied by the current neighboring time series TS n Feature-time sample TS n The value of [f][t] is consistent with the current time series TS i Feature-time sample TS i The difference between the values of [f][t].
[0214] In Table 1, the selection of random values in subprocess 360 is again represented by the rdm() function. It is worth noting that in subprocess 360, the range of random values is set to be high so that the synthetic feature-time sample TS s The resulting value of [f][t] will be closer to the current neighboring time series TS n Feature-time sample TS n The value of [f][t] (the current neighboring time series belongs to the minority class, as determined in subprocess 355) rather than being closer to the current time series TS i Feature-time sample TS i [f][t] value. For example, in Table 1, the range of random values is set to 0.9 to 1.0, which means that in the feature-time sample TS i [f][t] value and features - time sample TS n The value of [f][t] is within a range of possible values, ensuring that the synthetic feature-time sample TS s The resulting value of [f][t] is in the feature-time sample TS n [f][t] is within the first 10% of the nearest value. As another example, if the range of random values is set to 0.7 to 0.9, the synthetic feature-time sample TS is guaranteed to bes The resulting value of [f][t] is in the feature-time sample TS n [f][t] is outside the top 10% of the nearest values, but is in the feature-time sample TS n The random value is within the first 30% of the nearest values of [f][t]. In other words, the random value acts as a scaling factor to shrink the feature-time sample TS. i [f][t] and neighboring features-time samples TS n The distance between the values of [f][t]. It should be understood that the range of random values 0.9 to 1.0 is only an example, and other ranges can be used (e.g., 0.8 to 1.0, 0.7 to 0.9, etc.). In general, the range should be, but need not be, a subset of the values between 0.5 and 1.0, so that the random values are guaranteed to be closer to 1.0 than to 0.0.
[0215] In subprocess 365 (eg, corresponding to row 17 in Table 1), based on the current time series TS under consideration, i Feature-time sample TS i The value of [f][t] and the current neighboring time series TS under consideration n Feature-time sample TS n The values of [f][t] are used to generate synthetic features-time samples TS s [f][t] value. For example, as shown in Table 1, the synthetic feature-time sample TS s The value of [f][t] can be calculated as the current time series TS i Feature-time sample TS i The value of [f][t] is added to the product of a random value from a set of possible values multiplied by the current neighboring time series TS n Feature-time sample TS n The value of [f][t] is consistent with the current time series TS i Feature-time sample TS i The difference between the values of [f][t].
[0216] In Table 1, the selection of random values in subprocess 365 is again represented by the rdm() function. However, in contrast to subprocess 360, in subprocess 365, the range of random values is set to be low so that the synthetic feature-time sample TS s The result value of [f][t] will be closer to the current time series TS i Feature-time sample TS i The value of [f][t] is closer to the current neighboring time series TS n Feature-time sample TS n[f][t] value. For example, in Table 1, the range of random values is set to 0.0 to 0.1, which means that in the feature-time sample TS i [f][t] value and features - time sample TS n The value of [f][t] is within a range of possible values, ensuring that the synthetic feature-time sample TS s The resulting value of [f][t] is in the feature-time sample TS i [f][t] is within the top 10% of the nearest values. It should be understood that the range of random values 0.0 to 0.1 is just one example, and other ranges (e.g., 0.0 to 0.2, 0.1 to 0.3, etc.) can be used. In general, the range should be, but need not be, a subset of the values between 0.0 and 0.5, so that the random values are guaranteed to be closer to 0.0 than to 1.0.
[0217] It should be understood that the generation of the composite values described with respect to subprocesses 350, 360, and 365 is merely an example, and that the composite values may be generated in other ways. For example, in an alternative embodiment, the composite values may all be generated by adding random numbers to the current time series TS i Feature-time sample TS i [f][t] is generated. In this case, when the feature-time sample TS i When [f][t] is in the minority category, it can generate feature-time samples TS i [f][t] is not in the minority class. As another alternative, synthetic values can be generated based on a threshold. For example, when the feature-time sample TS i When [f][t] is in the minority class, a synthetic value can be randomly generated so that it is within a certain distance from the feature-time sample TS. i [f][t] within the first threshold distance and / or away from the current neighboring time series TS n Feature-time sample TS n [f][t] is at least the second threshold distance. Conversely, when the feature-time sample TS i [f][t] is not in the minority class and feature-time sample TS n When [f][t] is in the minority class, a synthetic value can be randomly generated so that it is within a certain distance from the feature-time sample TS. n [f][t] first threshold distance within and / or away from the feature - time sample TS i [f][t] is at least a second threshold distance. More generally, when the feature-time sample TS i When [f][t] is in the minority class, a synthetic value can be generated according to the first bias toward the minority class, and when the feature-time sample TS iWhen [f][t] is not in the minority class, a composite value is generated based on a second bias that is less biased toward the minority class than the first bias (e.g., biased away from the minority class, biased toward the majority class, biased toward the minority class but less than the first bias, etc.).
[0218] In subprocess 370 (eg, corresponding to row 22 in Table 1), the new synthetic time series TS s Add to the synthetic dataset (e.g., X in Table 1 s ). Alternatively, the new synthetic time series TS s directly added to the original data set (e.g., X in Table 1). Subprocess 370 may include adding the representation of the synthetic time series TS s The label of the category to which the synthetic time series belongs is added to the synthetic time series (eg, corresponding to row 21 in Table 1). In an embodiment, the label may be a label indicating the synthetic time series TS s and create synthetic time series TS from it s Time series TS i and the adjacent time series TS n For example, in Table 1, for a binary category, the synthetic time series TS n The label is calculated as the sum of the current time series TS i The label is multiplied by the synthetic time series TS s With the current adjacent time series TS n The product of the distances between them, and the current neighboring time series TS n The label is multiplied by the synthetic time series TS s With the current time series TS i The product of the distances between and divided by the sum of the following two: the synthetic time series TS s With the current time series TS i The distance between them and the synthetic time series TS s With the current adjacent time series TS n The distance between the time series can be calculated using any distance metric, including the same distance metric discussed above for identifying the k nearest neighbors in subprocess 315. It should be understood that in binary classification, the time series TS i and TS n The value of a given label can be either 0 or 1. However, the synthetic time series TS s The value of the label can be a real value, which represents the synthetic time series TS sThe probability of the class into which it should be classified. In alternative embodiments, the values of labels in binary classification can be binarized as either the minority class or the majority class (e.g., by rounding real values to produce 0 or 1). In such embodiments, the following line can be added after line 21 in Table 1 to binarize the class labels using a binarization threshold (e.g., binarization_threshold), which can be user-specified (e.g., included as an input to algorithm 300) or system-specified:
[0219]
[0220] In subprocess 375 (eg, corresponding to row 27 in Table 1), the synthetic data set (eg, X s ) is output as the product of algorithm 300. In an alternative embodiment, the synthesized data set may be directly incorporated into the original data set.
[0221] In summary, the upsampling algorithm 300 can sample time series from the original data set, and then iterate through the pairing of each time series with one of its k nearest neighbors belonging to the minority class to create a synthetic time series. The use of the neighborhood and distance metrics of the k nearest neighbors involves the temporal correlation between the time samples, because it ensures that the synthetic time samples will not be too far away from the original samples. For each feature of each time sample, a synthetic feature value of the synthetic time sample is generated. When possible, each synthetic feature value is calculated to be closer to the feature value of the time sample belonging to the minority class. In other words, the synthetic feature value imitates the feature value of the time sample belonging to the minority class. As a result, the synthetic time series created from these synthetic feature values imitates the time series belonging to the minority class. Then, these synthetic time series can be combined into the training data set to improve the ratio of the number of time series belonging to the minority class to the total number of time series and / or the number of time series belonging to the majority class. It is worth noting that the upsampling algorithm 300 is able to reuse the already available time series data, so that there is no need to obtain new time series data, and the desired ratio can be achieved while retaining the entire original data set (e.g., without downsampling).
[0222] The baseband system 260 is also communicatively coupled to the processor(s) 210. The processor(s) 210 may access data storage areas 215 and 220. The processor(s) 210 are preferably configured to execute instructions (i.e., computer programs, such as the disclosed software) which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. When executed, such computer programs may enable the system 200 to perform various functions of the disclosed embodiments.
[0223] 2.3. Downsampling Algorithm
[0224] The goal of downsampling is to reduce the severity of class imbalance in a dataset by removing data representing the majority class. In the case of time series data, a subset of time series belonging to the majority class can be removed from the dataset, thereby adjusting the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class.
[0225] Figure 4 The downsampling algorithm according to an embodiment is illustrated. The downsampling algorithm 400 may be implemented as a software module hosted and / or executed on the platform 110 or the user system 130 for managing training data for a machine learning model, training a machine learning model, deploying a machine learning model, and / or executing a machine learning model. The downsampling algorithm 400 may represent a module of the entire resampling algorithm or an overall resampling algorithm (e.g., combined with the upsampling algorithm 300).
[0226] In subprocess 410, algorithm 400 may receive a data set to be downsampled. In addition, algorithm 400 may receive one or more hyperparameters, such as a threshold and a tolerance value. The threshold represents the ideal or expected ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class. In an embodiment, the threshold is 1.0. The tolerance value represents an acceptable difference between the actual ratio and the expected ratio. In other words, the goal of algorithm 400 is to adjust the data set so that the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class is greater than the threshold minus the tolerance value and less than the threshold plus the tolerance value. As an example, the tolerance value may be 0.01. In general, the tolerance value may be any scalar value suitable for a given task, and will typically be in the range of 0.001 to 0.01.
[0227] In subprocess 420, it may be determined whether the absolute value of the difference between the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class and the threshold value is less than the tolerance value. When the majority of the time samples of the time series are labeled with the minority class, the time series may belong to the minority class. Similarly, when the majority of the time samples of the time series are labeled with the majority class, the time series may belong to the majority class. If the absolute value is not less than the tolerance value (i.e., "No" in subprocess 420), algorithm 400 proceeds to subprocess 430. Otherwise, if the absolute value is less than the tolerance value (i.e., "Yes" in subprocess 420), algorithm 400 proceeds to subprocess 450.
[0228] In subprocess 430, time series belonging to the majority category are sampled from the data set. In an embodiment, the sampled time series are randomly selected from the set of time series belonging to the majority category in the data set. In addition, the data set may be shuffled before selecting the time series. However, it should be understood that the time series may be selected in any different manner.
[0229] In subprocess 440, the sampled time series may be removed from the data set. Algorithm 400 may then return to subprocess 420 to re-evaluate the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class. In an alternative embodiment, subprocess 430 may sample the number of time series to be removed so that the ratio is within a tolerance value from a threshold, and subprocess 440 may remove all sampled time series at once, so that subprocesses 420-440 need only be performed once.
[0230] In subprocess 450, the data set may be output. It will be appreciated that the size of the output data set may be reduced by the number of time series removed in one or more iterations of subprocess 440. Due to the removal of those time series belonging to the majority class, the output data set may be more balanced than the input data set. For example, in the case of a binary class, the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class in the output data set will be closer to 1.0 than the ratio in the input data set.
[0231] The baseband system 260 is also communicatively coupled to the processor(s) 210. The processor(s) 210 may access data storage areas 215 and 220. The processor(s) 210 are preferably configured to execute instructions (i.e., computer programs, such as the disclosed software) which may be stored in the main memory 215 or the secondary memory 220. Computer programs may also be received from the baseband processor 260 and stored in the main memory 210 or the secondary memory 220, or executed upon receipt. When executed, such computer programs may enable the system 200 to perform various functions of the disclosed embodiments.
[0232] 2.4. Example resampling process
[0233] In an embodiment, the resampling process includes both the upsampling algorithm 300 and the downsampling algorithm 400. Alternatively, the resampling process may include the upsampling algorithm 300 but not the downsampling algorithm 400. As yet another alternative, the resampling process may include the downsampling algorithm 400 but not the upsampling algorithm 300.
[0234] Figure 5 The diagram illustrates a resampling process for reducing class imbalance in a training dataset for machine learning using both upsampling and downsampling according to an embodiment. The process 500 may be implemented as a software module hosted and / or executed on the platform 110 or the user system 130 for managing training data for a machine learning model, training a machine learning model, deploying a machine learning model, and / or executing a machine learning model.
[0235] In subprocess 510, process 500 may receive a data set to be resampled. In addition, process 500 may receive one or more hyperparameters, such as any of the hyperparameters described with respect to algorithms 300 and 400. For example, these hyperparameters may include the number of time series to be sampled from the original data set, the number of synthetic time series to be generated from each sampled time series in the original data set, and the number of nearest neighbors to be used, as described with respect to algorithm 300. These hyperparameters may also include threshold and tolerance values, as described with respect to algorithm 400.
[0236] In subprocess 520, it may be determined whether the absolute value of the difference between the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class in the data set received in subprocess 510 and the threshold is less than the tolerance value. Subprocess 520 may be the same as or similar to subprocess 420, and therefore, any description of subprocess 420 is also applicable to subprocess 520, and vice versa. If the absolute value is not less than the tolerance value (i.e., "No" in subprocess 520), process 500 may execute upsampling algorithm 300. Otherwise, if the absolute value is less than the tolerance value (i.e., "Yes" in subprocess 520), process 500 may continue to subprocess 550.
[0237] As discussed elsewhere herein, the upsampling algorithm 300 can generate a synthetic time series based on the data set received in subprocess 510. In essence, the upsampling algorithm 300 generates a synthetic time series by computing the average information from neighboring time series, thereby reducing temporal bias through randomization and convex combination techniques. The upsampling algorithm 300 ensures that there are enough time series representing the minority class to properly balance a severely imbalanced data set.
[0238] In subprocess 530, the synthetic data set including the synthetic time series generated by the upsampling algorithm 300 may be combined with the original data set to generate an enhanced data set. In an alternative embodiment, the upsampling algorithm 300 may combine the synthetic time series into the original data set when generating it to directly generate the enhanced data set, so that subprocess 530 may be omitted.
[0239] Process 300 may input the augmented dataset into a downsampling algorithm 400 to correct any residual class imbalance. As discussed elsewhere herein, downsampling algorithm 400 may remove time series belonging to the majority class until the ratio of the number of time series belonging to the minority class to the number of time series belonging to the majority class is within a tolerance of a threshold. Downsampling algorithm 400 provides fine control over this ratio to balance the augmented dataset to a predefined accuracy ratio represented by a tolerance value.
[0240] It should be understood that if the upsampling algorithm 300 produces a certain number of synthetic time series so that the ratio increases to within the tolerance range of the threshold, the downsampling algorithm 400 can be completed without removing any time series belonging to the majority category. In other words, if the first iteration of the subprocess 420 produces a "yes" determination, the algorithm 400 will execute the subprocesses 410, 420 and 450 without executing any iterations of the subprocesses 430 and 440.
[0241] In subprocess 550, an enhanced and potentially downsampled dataset is output. It should be understood that the output dataset will have been rebalanced to within a tolerable class imbalance level. The output dataset can then be used to train a machine learning model, such as a binary classifier. Since the training dataset is more balanced, the resulting machine learning model can more accurately identify time series belonging to a minority class, which can represent rare events in mission-critical applications.
[0242] 3. Example Use Case
[0243] It should be understood that the processes described herein (including algorithm 300, algorithm 400, and resampling process 500) can be used to resample a data set for any machine learning model that classifies any type of time series in any application or context. However, in order to provide a specific, non-limiting example that demonstrates the capabilities of the disclosed process, a specific application will be described in detail. It should be understood that similar results can be achieved using the disclosed embodiments in other applications.
[0244] For demonstration purposes, a resampling process 500 is used to rebalance a training data set for a machine learning algorithm for time domain protection. Time domain protection represents a mission-critical application in power systems. In time domain protection, the task is to train a machine learning model to classify features of an electrical system for fault detection, phase selection, reach, and / or the like. Due to the problem definition, the training data set may be unbalanced. For example, an overhead line fault may have a very high or very low coverage setting. Assuming that the fault locations in the original training data set are evenly distributed, a 10% coverage setting will cause only 10% of all time series in the original training data set to be labeled with a fault class. In this case, it should be understood that the fault or trip class (e.g., indicating a fault that requires a circuit breaker to trip) is a minority class. The majority class may be a non-fault or limit class (e.g., indicating a fault that is not faulty or does not require a circuit breaker to trip). Alternatively, for a high coverage setting, the fault or trip class may be the majority class, while the non-fault or limit class may be the minority class. Therefore, it should be understood that the coverage setting will generally determine which class is the majority class and which class is the minority class in the training data set.
[0245] Unlike IID data, time series data is three-dimensional, with correlated time samples along the time dimension. As a result, class imbalance is amplified in time series data, making conventional resampling techniques (such as bootstrapping) prohibitively expensive in terms of processing time and memory requirements. In contrast to these conventional resampling techniques, the disclosed resampling embodiments are consistent across correlated time samples, efficiently achieve rebalancing within a given tolerance (which may vary depending on the specific application), and preserve the temporal correlation of time samples by considering the entire sequence of features in the time series when inputting a synthetic time series during upsampling.
[0246] In addition, the disclosed embodiments facilitate the application of machine learning to contexts with variable settings. For example, in the context of time domain protection, a machine learning model can be trained to classify time series into one of two categories that specify either a trip decision (e.g., trip a circuit breaker) or a limit decision (e.g., do not trip a circuit breaker). The decision is affected by a given set of coverage settings, which makes the class prediction and resulting decision more challenging than in a context with fixed coverage settings.
[0247] Precision and recall (as defined below) are two metrics that can be used to capture the strength of a trained machine learning model:
[0248]
[0249] If class imbalance occurs in the training dataset, there is the potential to give unequal importance to false positives and false negatives. Therefore, a good candidate for capturing the predictive accuracy of a machine learning classifier is F. β When β=1, this score is also called F1 score. β The score is a summary of the harmonic mean of precision and recall and is given by:
[0250]
[0251] During experiments with the disclosed resampling techniques, an attention mechanism was used to capture the relative importance of past information in the current training. The specific attention mechanism used is described in Thang et al., "Effective Methods for Attention-Based Neural Machine Translation" (arXiv abs / 1508.04025 (2015), which is hereby incorporated by reference herein as if fully set forth). Different machine learning models were trained with non-causal attention (i.e., where all time samples are used to learn attention weights), causal attention (i.e., where future time samples are masked when calculating the attention weight at a certain moment, and all time samples up to that moment are used to calculate the attention weight), and no attention mechanism. Multiple machine learning models were also trained with and without the disclosed resampling techniques and with different hyperparameters.
[0252] The experiments demonstrate that the prediction accuracy of the machine learning models improves when trained on a dataset that has been resampled according to the disclosed resampling technique to reduce class imbalance, as measured by the F1 score. In particular, three machine learning models were trained for fault detection in time domain protection: (1) a fixed line length of 200 kilometers with a coverage setting within 130-190 kilometers; (2) a variable line length with a coverage setting within 120-180 kilometers; and (3) a variable line length with a coverage setting within 50-120 kilometers. For comparison, each machine learning model was independently trained using both a non-resampled training dataset and a training dataset resampled according to the disclosed resampling technique. The training dataset included time series created from a line with one tower having a fixed line length and time series created from lines with two towers having different line lengths. The F1 scores of the trained machine learning models are depicted in Table 2 below:
[0253]
[0254] Figure 6 A comparison between precision-recall curves obtained for a machine learning model trained separately without a resampled dataset and with a dataset resampled according to the disclosed resampling technique is illustrated. It is noteworthy that without the disclosed resampling of the training dataset, the precision of the resulting machine learning model is significantly lower and the false positives are relatively higher.
[0255] As demonstrated by the above results, the disclosed resampling technique for reducing class imbalance in time series data can improve machine learning classification of rare events. In power systems, rare events can occur in the form of faults in transmission lines. Therefore, the disclosed resampling technique can be used to improve the prediction accuracy of distance protection and time domain protection.
[0256] In addition to protection tasks, class imbalances in time series data can also occur in the form of cascading failures in transmission networks, in which rare combinations of conditions, uncommon events, and / or inadequate countermeasures can lead to further line disconnections, ultimately causing the fault to spread. In such contexts, balancing the class labels in the training dataset can produce better estimates and prevent cascading failures, which are rare but have a huge impact. Another context in which the disclosed resampling techniques can be particularly useful is load forecasting based on time series data. In load forecasting, rare load events (such as lightning, heat waves, and / or the like) can create unbalanced datasets. In such cases, the dataset can be resampled using (multiple) disclosed processes before training a machine learning model to perform load forecasting. Other potential contexts include resampling training datasets for machine learning models that classify time series data into categories of: outage detection, congestion, circuit breaker status (e.g., open or closed), normal or rare event surge operation status (e.g., external fault, internal fault such as bushing or explosion), transformer status (e.g., surge, normal, internal fault, or external fault ride-through), line fault status (e.g., AG, BG, CG, AB, BC, or CA), anomalies or outliers in Supervisory Control and Data Acquisition (SCADA) data, etc. In general, any time series dataset can be resampled according to the disclosed resampling techniques, especially in the case of binary classification.
[0257] Another example where resampling can be particularly useful is estimating the probability of an unexpected event occurring in a transmission network. The topology of a transmission substation can change due to actions taken by operators to maintain the facility and mitigate adverse operating conditions (such as high circuit loads, over-limit voltages, forced outages, and / or the like). In some cases, changes in the topology may result in a situation where a protection system operates in response to a fault occurring on the network to remove two or more elements while clearing the fault. Such topologies significantly increase the risk level of the network because they expose the network to multiple power outage unexpected events due to a single fault, and because the risk associated with a particular unexpected event is the expected value of the consequences of that unexpected event. Therefore, estimating the risk of higher-order unexpected events is important for the safety of the power system, and it is necessary to estimate the probabilities of these unexpected events. Therefore, the disclosed resampling techniques can be applied to time series data used to train machine learning models to estimate the probabilities of these unexpected events.
[0258] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles described herein may be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, it will be understood that the description and drawings presented herein represent currently preferred embodiments of the present disclosure, and therefore represent the subject matter broadly contemplated by the present disclosure. It should be further understood that the scope of the present disclosure fully encompasses other embodiments that may become apparent to those skilled in the art, and the scope of the present disclosure is accordingly not limited.
[0259] The combinations described herein (such as, "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof") include any combination of A, B, and / or C, and may include multiple A's, multiple B's, or multiple C's. Specifically, phrases such as "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B, and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof" may be only A, only B, only C, A and B, A and C, B and C, or A, B, and C, and any such combination may include one or more members of its components A, B, and / or C. For example, the combination of A and B may include one A and multiple B's, multiple A's and one B's, or multiple A's and multiple B's.
Claims
1. A computer-implemented method for detecting an event in an electrical system, the method comprising executing, using at least one hardware processor, a process to: Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps: receiving at least a subset of the training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories, For each time series in the at least subset of the training data set, identifying a number of neighboring time series in the at least a subset of the training data set based on a distance metric, for each of one or more of the neighboring time series, for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and generating a synthetic time series from the synthetic time samples comprising the synthetic values of the one or more features, and Adding the generated synthetic time series to the training dataset; After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset; executing a trained machine learning algorithm on the acquired data related to the electrical system; as well as The event is detected based on execution of the trained machine learning algorithm.
2. The method according to claim 1, wherein: The event is an anomaly in the electrical system, in particular a fault in at least one transmission line of the electrical system.
3. The method according to claim 1 or 2, further comprising: After adding the generated synthetic time series to the training dataset, and before training the machine learning algorithm, removing one or more majority time series from the training dataset, wherein a majority of the plurality of labeled time samples include labels that do not represent a minority class.
4. The method according to claim 3, wherein: The removal is performed until a maximum number of iterations is reached.
5. The method according to claim 3, wherein: The removing is performed until a ratio between the number of minority time series in the training dataset and the number of majority time series in the training dataset is within a tolerance of a threshold, wherein a minority time series is defined as a time series in which a majority of the plurality of labeled time samples include a label representing the minority class.
6. The method according to any one of claims 1 to 5, wherein: Receiving at least a subset of the training data set includes selecting a random sample including a predefined number of time series from the training data set.
7. The method according to any one of claims 1 to 6, wherein: Receiving at least a subset of the training data set includes selecting a sample including a predefined number of time series from the training data set to include a higher proportion of minority time series than majority time series relative to the training data set.
8. The method according to any one of claims 1 to 7, wherein: The plurality of categories includes two categories.
9. The method according to any one of claims 1 to 8, wherein: The number of neighboring time series is a predefined number greater than one.
10. The method according to any one of claims 1 to 9, further comprising: Using the at least one hardware processor, a predefined number of synthetic time series are generated for each time series in the at least a subset of the training data set.
11. The method according to claim 10, wherein: Generating a predefined number of synthetic time sequences includes randomly selecting one of the identified number of adjacent time sequences for generating the synthetic time sequence until the predefined number of synthetic time sequences have been generated.
12. The method according to any one of claims 1 to 11, wherein: For each marked time sample in the plurality of marked time samples in the adjacent time series, and for each feature in the one or more features, generating a composite value of the feature includes: When the label of the corresponding labeled time sample in the time series represents a minority class among the plurality of classes, generating the synthetic value of the feature according to a first bias toward the minority class, and When the label of the corresponding labeled time sample in the time series does not represent the minority class, the synthetic value of the feature is generated according to a second bias that is less biased toward the minority class than the first bias.
13. The method according to claim 12, wherein: Generating the composite value of the feature according to the first bias includes adding a randomly generated value to the value of the feature in the corresponding labeled time sample in the time series.
14. The method according to claim 12, wherein: Generating the synthetic value of the feature according to the second bias includes: calculating a distance between the value of the feature in the corresponding labeled time sample in the time series and the value of the feature in the labeled time sample in an adjacent time series; Randomly generate scaling factors; calculating a scaled value by multiplying the calculated distance by the randomly generated scaling factor; and The scaled value is added to the value of the feature in the corresponding labeled time sample in the time series.
15. The method according to claim 14, wherein: When the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series represents the minority class, limiting the range of the scaling value to produce a scaled value that is closer to the value of the feature in the labeled time sample in the adjacent time series than to the value of the feature in the corresponding labeled time sample in the time series.
16. The method according to claim 14, wherein: When the label of the corresponding labeled time sample in the time series does not represent the minority class and the label of the labeled time sample in the adjacent time series does not represent the minority class, limiting the range of the scaling value to produce a scaled value that is closer to the value of the feature in the corresponding labeled time sample in the time series than to the value of the feature in the labeled time sample in the adjacent time series.
17. The method according to any one of claims 1 to 16, wherein: Generating the synthetic time series includes: marking the synthetic time series with a label value, wherein the label value is derived based on a label indicating that the time series is classified into one of multiple categories and a label indicating that the adjacent time series is classified into one of multiple categories.
18. The method according to claim 17, wherein: The label value is further derived based on a distance between the synthetic time series and the time series and a distance between the synthetic time series and the neighboring time series.
19. The method according to claim 18, wherein: The plurality of categories include two categories, and the label value is derived by binarizing a value based on the distance between the synthetic time series and the time series and the distance between the synthetic time series and the neighboring time series.
20. The method according to any one of claims 1 to 19, wherein: The one or more features represent parameters of an electrical system, and wherein at least one of the plurality of categories represents an action to be taken in response to an anomaly in the electrical system.
21. A system comprising: at least one hardware processor; Memory for storing software; as well as The software is configured to perform the following operations when executed by the at least one hardware processor: Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps: receiving at least a subset of the training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories, For each time series in the at least subset of the training data set, identifying a number of neighboring time series in the at least a subset of the training data set based on a distance metric, for each of one or more of the neighboring time series, for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and generating a synthetic time series from the synthetic time samples comprising the synthetic values of the one or more features, and Adding the generated synthetic time series to the training dataset; After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset; executing the trained machine learning algorithm on the acquired data related to the electrical system; as well as The event is detected based on execution of the trained machine learning algorithm.
22. A non-transitory computer readable medium having stored thereon instructions, wherein: The instructions, when executed by a processor, cause the processor to: Reduce class imbalance in the training dataset for a machine learning algorithm by following these steps: receiving at least a subset of the training data set, the training data set comprising a plurality of time series, each of the plurality of time series comprising a plurality of labeled time samples, each of the plurality of labeled time samples comprising a value of each of one or more features at a particular time within a time frame of the time series, and a label indicating that the time sample has been classified into one of a plurality of categories, For each time series in the at least subset of the training data set, identifying a number of neighboring time series in the at least a subset of the training data set based on a distance metric, for each of one or more of the neighboring time series, for each of the plurality of labeled time samples in the adjacent time series and for each of the one or more features, generating a composite value of the feature in a composite time sample, and generating a synthetic time series from the synthetic time samples comprising the synthetic values of the one or more features; and Adding the generated synthetic time series to the training dataset; After adding the generated synthetic time series to the training dataset, training the machine learning algorithm using at least a portion of the training dataset; executing the trained machine learning algorithm on the acquired data related to the electrical system; as well as The event is detected based on execution of the trained machine learning algorithm.
Citation Information
Patent Citations
Deep learning-based point-by-point classification fault detection method
CN113537360A
Computer-Implemented System And Method For Relational Time Series Learning
US20170154282A1
Anomaly detection in multidimensional time series data
US20190147300A1
