Method, system, and computer program product for classifying network activity based on classification-specific data patterns

Abstract

Described are a system, method, and computer program product for classifying network activity based on classification-specific data patterns. The method includes receiving training data associated with historic network activity. The method includes training a supervised learning model to output a classification of a plurality of classifications associated with network activity. The method includes training an unsupervised learning model to output an outlier score associated with each classification. The method includes receiving activity data in a subsequent time period and determining outlier scores for the activity data. Determining the outlier scores includes inputting the activity data to the trained unsupervised learning model and determining the outlier scores based on the trained unsupervised learning model. The method includes, in response to each outlier score satisfying a threshold, generating a classification of the activity data based on the trained supervised learning model.

Description

METHOD, SYSTEM, AND COMPUTER PROGRAM PRODUCT FOR CLASSIFYING NETWORK ACTIVITY BASED ON CLASSIFICATION-SPECIFIC DATA PATTERNSBACKGROUND1 . Technical Field

[0001] This disclosure relates generally to network monitoring, and, in some nonlimiting embodiments or aspects, to systems, methods, and computer program products for classifying network activity based on classification-specific data patterns to determine classification reliability.2. Technical Considerations

[0002] Classification models may be used to classify new data as belonging to one or more classifications (e.g., classes, groupings, etc.), based on patterns learned from historical data. However, classification model output may be inaccurate or difficult to interpret based on the patterns of the new data, particularly for patterns that might differ from the historical data. Furthermore, classification models may be biased based on data imbalances, where some classifications have disproportionately more data points for training, which may result in higher rates of false positives or false negatives for classifications with fewer training data points. Additionally, while a classification model may be accurate for one time period, data patterns for a given classification may change in a subsequent time period, making output classifications inaccurate over time. Numerous computational inefficiencies are associated with such inaccurate and biased classification models, such as wasted computing resources (e.g., memory, bandwidth, processing time, etc.) expended in responding to false positives, more computing resources required to train a sufficiently accurate model, and slower reaction time to detect that a classification model has become unreliable for continued use. Such inefficiencies are especially troublesome for classification models that are configured to classify network activity (e.g., transactions, user access to computing resources, communications, etc.), so that a modeling system can react accordingly (e.g., respond to anomalous activity, disable nefarious user accounts, etc.).

[0003] There is a need in the art for a technical solution to better predict classifications for new data, which is resilient to data imbalance, data bias, and shifting data patterns over time. There is a further need in the art for a technical solution toassess the reliability of the output of classification models, both for the use of generating classifications and determining when a model may need to be retrained.SUMMARY

[0004] According to some non-limiting embodiments or aspects, provided are systems, methods, and computer program products for classifying network activity based on classification-specific data patterns that overcome some or all of the deficiencies identified above.

[0005] According to some non-limiting embodiments or aspects, provided is a computer-implemented method for classifying network activity based on classificationspecific data patterns. The method includes receiving, with at least one processor, training data in a first time period associated with historic network activity. The method also includes training, with at least one processor, a supervised learning model based on the training data to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity. The method further includes training, with at least one processor, an unsupervised learning model based on the training data to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity. The method further includes receiving, with at least one processor, activity data in a second time period subsequent the first time period associated with network activity. The method further includes determining, with at least one processor, a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications. Determining the plurality of outlier scores includes inputting the activity data to the trained unsupervised learning model and determining the plurality of outlier scores based on at least one output of the trained unsupervised learning model. The method further includes comparing, with at least one processor, each outlier score of the plurality of outlier scores to at least one threshold. The method further includes, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generating, with at least one processor, a classification of the activity data based on the trained supervised learning model.

[0006] In some non-limiting embodiments or aspects, the at least one threshold may include a plurality of thresholds. The method may also include determining, withat least one processor, a plurality of candidate thresholds. The method may further include determining, with at least one processor, a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations includes a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination includes a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications. The method may further include generating, with at least one processor, a plurality of training scores based on the trained unsupervised learning model and the training data. The method may further include determining, with at least one processor, a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores. The method may further include determining, with at least one processor, a plurality of maximum thresholds based on the plurality of performance metrics. The method may further include determining, with at least one processor, the plurality of thresholds based on the plurality of maximum thresholds.

[0007] In some non-limiting embodiments or aspects, generating the plurality of training scores based on the trained unsupervised learning model and the training data may include inputting a plurality of data points of the training data to the trained unsupervised learning model, receiving a plurality of outputs from the trained unsupervised learning model based on the plurality of data points, and generating each training score of the plurality of training scores based on an output of the plurality of outputs.

[0008] In some non-limiting embodiments or aspects, each performance metric of the plurality of performance metrics may be associated with a pairing of each combination of the plurality of combinations. Determining the plurality of performance metrics may include, for each performance metric of the plurality of performance metrics, determining a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0009] In some non-limiting embodiments or aspects, the plurality of performance metrics may include a user-selected performance metric. Determining the plurality of maximum thresholds based on the plurality of performance metrics may include comparing each performance metric of the plurality of performance metrics to a threshold score, determining a subset of performance metrics of the plurality ofperformance metrics including performance metrics that satisfy the threshold score, and determining the plurality of maximum thresholds based on the subset of performance metrics.

[0010] In some non-limiting embodiments or aspects, the method may include repeating, over a plurality of time periods, a series of steps. The series of steps may include receiving, with at least one processor, new activity data in a new time period of the plurality of time periods associated with new network activity. The series of steps may also include determining, with at least one processor, a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications. The series of steps may further include comparing, with at least one processor, each new outlier score of the plurality of new outlier scores to the at least one threshold. The series of steps may further include, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determining, with at least one processor, that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable. The method may further include, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, updating, with at least one processor, the trained supervised learning model.

[0011] In some non-limiting embodiments or aspects, updating the trained supervised learning model may include receiving, with at least one processor, new training data based on network activity over the plurality of time periods, and training, with at least one processor, the supervised learning model based on the new training data.

[0012] In some non-limiting embodiments or aspects, the classification of the activity data may be associated with a type of anomalous network activity. The method may further include performing, with at least one processor, at least one remediative action based on the classification of the activity data, wherein the at least one remediative action includes: transmitting at least one alert to a computing device of a user; disabling at least one user account associated with the activity data; changing permissions of at least one user account associated with the activity data; or any combination thereof.

[0013] According to some non-limiting embodiments or aspects, provided is a computer-implemented method for classifying network activity based on classificationspecific data patterns. The system includes at least one processor programmed or configured to receive training data in a first time period associated with historic network activity. The at least one processor is also programmed or configured to train a supervised learning model based on the training data to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity. The at least one processor is further programmed or configured to train an unsupervised learning model based on the training data and to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity. The at least one processor is further programmed or configured to receive activity data in a second time period subsequent the first time period associated with network activity. The at least one processor is further programmed or configured to determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications. When determining the plurality of outlier scores, the at least one processor is programmed or configured to input the activity data to the trained unsupervised learning model and determine the plurality of outlier scores based on at least one output of the trained unsupervised learning model. The at least one processor is further programmed or configured to compare each outlier score of the plurality of outlier scores to at least one threshold. The at least one processor is further programmed or configured to, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

[0014] In some non-limiting embodiments or aspects, the at least one threshold may include a plurality of thresholds. The at least one processor may be further programmed or configured to determine a plurality of candidate thresholds. The at least one processor may be further programmed or configured to determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations includes a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination includes a candidate threshold of the plurality of candidate thresholdsassociated with a classification of the plurality of classifications. The at least one processor may be further programmed or configured to generate a plurality of training scores based on the trained unsupervised learning model and the training data. The at least one processor may be further programmed or configured to determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores. The at least one processor may be further programmed or configured to determine a plurality of maximum thresholds based on the plurality of performance metrics. The at least one processor may be further programmed or configured to determine the plurality of thresholds based on the plurality of maximum thresholds.

[0015] In some non-limiting embodiments or aspects, each performance metric of the plurality of performance metrics may be associated with a pairing of each combination of the plurality of combinations. When determining the plurality of performance metrics, the at least one processor may be programmed or configured to, for each performance metric of the plurality of performance metrics, determine a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0016] In some non-limiting embodiments or aspects, the plurality of performance metrics may include a user-selected performance metric. When determining the plurality of maximum thresholds based on the plurality of performance metrics, the at least one processor may be programmed or configured to compare each performance metric of the plurality of performance metrics to a threshold score, determine a subset of performance metrics of the plurality of performance metrics including performance metrics that satisfy the threshold score, and determine the plurality of maximum thresholds based on the subset of performance metrics.

[0017] In some non-limiting embodiments or aspects, the at least one processor may be further programmed or configured to, repeat over a plurality of time periods, a series of steps. The series of steps may include receiving new activity data in a new time period of the plurality of time periods associated with new network activity. The series of steps may also include determining a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications. The series of steps may further include comparing eachnew outlier score of the plurality of new outlier scores to the at least one threshold. The series of steps may further include, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determining that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable. The at least one processor may be further programmed or configured to, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

[0018] In some non-limiting embodiments or aspects, when updating the trained supervised learning model, the at least one processor may be programmed or configured to receive new training data based on network activity over the plurality of time periods. The at least one processor may be further programmed or configured to train the supervised learning model based on the new training data.

[0019] According to some non-limiting embodiments or aspects, provided is a computer program product for classifying network activity based on classificationspecific data patterns. The computer program product includes at least one non- transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to receive training data in a first time period associated with historic network activity. The one or more instructions also cause the at least one processor to train a supervised learning model based on the training data to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity. The one or more instructions further cause the at least one processor to train an unsupervised learning model based on the training data to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity. The one or more instructions further cause the at least one processor to receive activity data in a second time period subsequent the first time period associated with network activity. The one or more instructions further cause the at least one processor to determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications. When determining the plurality of outlier scores, the at least one processor is programmed or configured to input the activity data to the trained unsupervised learning model, and determine the plurality of outlierscores based on at least one output of the trained unsupervised learning model. The one or more instructions further cause the at least one processor to compare each outlier score of the plurality of outlier scores to at least one threshold. The one or more instructions further cause the at least one processor to, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

[0020] In some non-limiting embodiments or aspects, the at least one threshold may include a plurality of thresholds. The one or more instructions may further cause the at least one processor to determine a plurality of candidate thresholds. The one or more instructions may further cause the at least one processor to determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations includes a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination includes a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications. The one or more instructions may further cause the at least one processor to generate a plurality of training scores based on the trained unsupervised learning model and the training data. The one or more instructions may further cause the at least one processor to determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores. The one or more instructions may further cause the at least one processor to determine a plurality of maximum thresholds based on the plurality of performance metrics. The one or more instructions may further cause the at least one processor to determine the plurality of thresholds based on the plurality of maximum thresholds.

[0021] In some non-limiting embodiments or aspects, each performance metric of the plurality of performance metrics may be associated with a pairing of each combination of the plurality of combinations. The one or more instructions that cause the at least one processor to determine the plurality of performance metrics may cause the at least one processor to, for each performance metric of the plurality of performance metrics, determine a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0022] In some non-limiting embodiments or aspects, the plurality of performance metrics may include a user-selected performance metric. The one or more instructionsthat cause the at least one processor to determine the plurality of maximum thresholds based on the plurality of performance metrics may cause the at least one processor to compare each performance metric of the plurality of performance metrics to a threshold score, determine a subset of performance metrics of the plurality of performance metrics including performance metrics that satisfy the threshold score, and determine the plurality of maximum thresholds based on the subset of performance metrics.

[0023] In some non-limiting embodiments or aspects, the one or more instructions may further cause the at least one processor to, repeat over a plurality of time periods, a series of steps. The series of steps may include receiving new activity data in a new time period of the plurality of time periods associated with new network activity. The series of steps may also include determining a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications. The set of steps may include comparing each new outlier score of the plurality of new outlier scores to the at least one threshold. The set of steps may include, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determining that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable. The one or more instructions may further cause the at least one processor to, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

[0024] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to update the trained supervised learning model may cause the at least one processor to receive new training data based on network activity over the plurality of time periods, and train the supervised learning model based on the new training data.

[0025] Other non-limiting embodiments or aspects will be set forth in the following numbered clauses:

[0026] Clause 1 : A computer-implemented method, comprising: receiving, with at least one processor, training data in a first time period associated with historic network activity; training, with at least one processor, a supervised learning model based on the training data, to produce a trained supervised learning model configured to outputat least one classification of a plurality of classifications associated with network activity; training, with at least one processor, an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receiving, with at least one processor, activity data in a second time period subsequent the first time period associated with network activity; determining, with at least one processor, a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein determining the plurality of outlier scores comprises: inputting the activity data to the trained unsupervised learning model; and determining the plurality of outlier scores based on at least one output of the trained unsupervised learning model; comparing, with at least one processor, each outlier score of the plurality of outlier scores to at least one threshold; and, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generating, with at least one processor, a classification of the activity data based on the trained supervised learning model.

[0027] Clause 2: The computer-implemented method of clause 1 , wherein the at least one threshold comprises a plurality of thresholds, the method further comprising: determining, with at least one processor, a plurality of candidate thresholds; determining, with at least one processor, a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generating, with at least one processor, a plurality of training scores based on the trained unsupervised learning model and the training data; determining, with at least one processor, a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determining, with at least one processor, a plurality of maximum thresholds based on the plurality of performance metrics; and determining, with at least one processor, the plurality of thresholds based on the plurality of maximum thresholds.

[0028] Clause 3: The method of clause 1 or clause 2, wherein generating the plurality of training scores based on the trained unsupervised learning model and thetraining data comprises: inputting a plurality of data points of the training data to the trained unsupervised learning model; receiving a plurality of outputs from the trained unsupervised learning model based on the plurality of data points; and generating each training score of the plurality of training scores based on an output of the plurality of outputs.

[0029] Clause 4: The method of any of clauses 1 -3, wherein each performance metric of the plurality of performance metrics is associated with a pairing of each combination of the plurality of combinations, and wherein determining the plurality of performance metrics comprises, for each performance metric of the plurality of performance metrics: determining a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0030] Clause 5: The method of any of clauses 1 -4, wherein the plurality of performance metrics comprises a user-selected performance metric; and wherein determining the plurality of maximum thresholds based on the plurality of performance metrics comprises: comparing each performance metric of the plurality of performance metrics to a threshold score; determining a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determining the plurality of maximum thresholds based on the subset of performance metrics.

[0031] Clause 6: The method of any of clauses 1 -5, further comprising: repeating over a plurality of time periods: receiving, with at least one processor, new activity data in a new time period of the plurality of time periods associated with new network activity; determining, with at least one processor, a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications; comparing, with at least one processor, each new outlier score of the plurality of new outlier scores to the at least one threshold; and, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determining, with at least one processor, that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; and, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, updating, with at least one processor, the trained supervised learning model.

[0032] Clause 7: The method of any of clauses 1 -6, wherein updating the trained supervised learning model comprises: receiving, with at least one processor, new training data based on network activity over a plurality of time periods; and training, with at least one processor, the supervised learning model based on the new training data.

[0033] Clause 8: The method of any of clauses 1 -7, wherein the classification of the activity data is associated with a type of anomalous network activity, the method further comprising: performing, with at least one processor, at least one remediative action based on the classification of the activity data, wherein the at least one remediative action comprises: transmitting at least one alert to a computing device of a user; disabling at least one user account associated with the activity data; changing permissions of at least one user account associated with the activity data; or any combination thereof.

[0034] Clause 9: A system comprising at least one processor programmed or configured to: receive training data in a first time period associated with historic network activity; train a supervised learning model based on the training data, to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity; train an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receive activity data in a second time period subsequent the first time period associated with network activity; determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein, when determining the plurality of outlier scores, the at least one processor is programmed or configured to: input the activity data to the trained unsupervised learning model; and determine the plurality of outlier scores based on at least one output of the trained unsupervised learning model; compare each outlier score of the plurality of outlier scores to at least one threshold; and, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

[0035] Clause 10: The system of clause 9, wherein the at least one threshold comprises a plurality of thresholds, and wherein the at least one processor is furtherprogrammed or configured to: determine a plurality of candidate thresholds; determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generate a plurality of training scores based on the trained unsupervised learning model and the training data; determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determine a plurality of maximum thresholds based on the plurality of performance metrics; and determine the plurality of thresholds based on the plurality of maximum thresholds.

[0036] Clause 1 1 : The system of clause 9 or clause 10, wherein each performance metric of the plurality of performance metrics is associated with a pairing of each combination of the plurality of combinations, and wherein, when determining the plurality of performance metrics, the at least one processor is programmed or configured to, for each performance metric of the plurality of performance metrics: determine a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0037] Clause 12: The system of any of clauses 9-1 1 , wherein the plurality of performance metrics comprises a user-selected performance metric; and wherein, when determining the plurality of maximum thresholds based on the plurality of performance metrics, the at least one processor is programmed or configured to: compare each performance metric of the plurality of performance metrics to a threshold score; determine a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determine the plurality of maximum thresholds based on the subset of performance metrics.

[0038] Clause 13: The system of any of clauses 9-12, wherein the at least one processor is further programmed or configured to: repeat over a plurality of time periods: receive new activity data in a new time period of the plurality of time periods associated with new network activity; determine a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of theplurality of classifications; compare each new outlier score of the plurality of new outlier scores to the at least one threshold; and, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; and, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

[0039] Clause 14: The system of any of clauses 9-13, wherein, when updating the trained supervised learning model, the at least one processor is programmed or configured to: receive new training data based on network activity over a plurality of time periods; and train the supervised learning model based on the new training data.

[0040] Clause 15: A computer program product comprising at least one non- transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive training data in a first time period associated with historic network activity; train a supervised learning model based on the training data, to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity; train an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receive activity data in a second time period subsequent the first time period associated with network activity; determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein, when determining the plurality of outlier scores, the at least one processor is programmed or configured to: input the activity data to the trained unsupervised learning model; and determine the plurality of outlier scores based on at least one output of the trained unsupervised learning model; compare each outlier score of the plurality of outlier scores to at least one threshold; and, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

[0041] Clause 16: The computer program product of clause 15, wherein the at least one threshold comprises a plurality of thresholds, and wherein the one or moreinstructions further cause the at least one processor to: determine a plurality of candidate thresholds; determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generate a plurality of training scores based on the trained unsupervised learning model and the training data; determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determine a plurality of maximum thresholds based on the plurality of performance metrics; and determine the plurality of thresholds based on the plurality of maximum thresholds.

[0042] Clause 17: The computer program product of clause 15 or clause 16, wherein each performance metric of the plurality of performance metrics is associated with a pairing of each combination of the plurality of combinations, and wherein the one or more instructions that cause the at least one processor to determine the plurality of performance metrics cause the at least one processor to, for each performance metric of the plurality of performance metrics: determine a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

[0043] Clause 18: The computer program product of any of clauses 15-17, the plurality of performance metrics comprises a user-selected performance metric; and wherein the one or more instructions that cause the at least one processor to determine the plurality of maximum thresholds based on the plurality of performance metrics cause the at least one processor to: compare each performance metric of the plurality of performance metrics to a threshold score; determine a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determine the plurality of maximum thresholds based on the subset of performance metrics.

[0044] Clause 19: The computer program product of any of clauses 15-18, wherein the one or more instructions further cause the at least one processor to: repeat over a plurality of time periods: receive new activity data in a new time period of the plurality of time periods associated with new network activity; determine a plurality of newoutlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications; compare each new outlier score of the plurality of new outlier scores to the at least one threshold; and, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; and, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

[0045] Clause 20: The computer program product of any of clauses 15-19, wherein the one or more instructions that cause the at least one processor to update the trained supervised learning model cause the at least one processor to: receive new training data based on network activity over a plurality of time periods; and train the supervised learning model based on the new training data.

[0046] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the present disclosure. As used in the specification and the claims, the singular form of “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise.BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Additional advantages and details of the disclosure are explained in greater detail below with reference to the exemplary embodiments or aspects that are illustrated in the accompanying schematic figures, in which:

[0048] FIG. 1 is a diagram of a non-limiting embodiment or aspect of an environment in which systems, devices, products, apparatus, and / or methods,described herein, may be implemented, according to non-limiting embodiments or aspects of the present disclosure;

[0049] FIG. 2 is a diagram of one or more components, devices, and / or systems, according to some non-limiting embodiments or aspects of the present disclosure;

[0050] FIG. 3 is a flowchart of a method for classifying network activity based on classification-specific data patterns, according to some non-limiting embodiments or aspects of the present disclosure;

[0051] FIG. 4 is a plurality of bar graphs demonstrating changes in data patterns over time, according to some non-limiting embodiments or aspects of the present disclosure;

[0052] FIG. 5 is a pair of scatterplots illustrating a detected unreliable classification of data, according to some non-limiting embodiments or aspects of the present disclosure;

[0053] FIG. 6 is a flowchart illustrating a process for data classification based on classification-specific data patterns, according to some non-limiting embodiments or aspects of the present disclosure;

[0054] FIG. 7 is a pair of scatterplots showing outlier scoring using an unsupervised learning model, according to some non-limiting embodiments or aspects of the present disclosure;

[0055] FIG. 8 is a scatterplot showing classification-based outlier scoring using an unsupervised learning model, according to some non-limiting embodiments or aspects of the present disclosure;

[0056] FIG. 9 is a pair of scatterplots showing use cases of classification-based outlier scoring using an unsupervised learning model, according to some non-limiting embodiments or aspects of the present disclosure;

[0057] FIG. 10A is a graphical representation of the relationship between outlier scores and performance metrics, according to some non-limiting embodiments or aspects of the present disclosure;

[0058] FIG. 10B is a graphical representation of the relationship between outlier scores and performance metrics, according to some non-limiting embodiments or aspects of the present disclosure; and

[0059] FIG. 1 1 is a graphical representation of determining outlier score thresholds based on performance metrics, according to some non-limiting embodiments or aspects.DETAILED DESCRIPTION

[0060] For purposes of the description hereinafter, the terms “upper”, “lower”, “right”, “left”, “vertical”, “horizontal”, “top”, “bottom”, “lateral”, “longitudinal,” and derivatives thereof shall relate to non-limiting embodiments or aspects as they are oriented in the drawing figures. However, it is to be understood that non-limiting embodiments or aspects may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0061] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. The phase “based on” may also mean “in response to” where appropriate.

[0062] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, and / or the like.

[0063] As used herein, the term “acquirer institution” may refer to an entity licensed and / or approved by a transaction service provider to originate transactions (e.g., payment transactions) using a payment device associated with the transaction serviceprovider. The transactions the acquirer institution may originate may include payment transactions (e.g., purchases, original credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term “acquirer system” may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0064] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0065] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit.

[0066] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0067] As used herein, the terms “electronic wallet” and “electronic wallet application” refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device executing an electronic wallet application, and may further include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An “electronic wallet provider” may include an entity that provides and / or maintains an electronic wallet for a customer, such as Google Pay®, Android Pay®, Apple Pay®, Samsung Pay®, and / or other like electronic payment systems. In some non-limiting examples, an issuer bank may be an electronic wallet provider.

[0068] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0069] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by oron behalf of a merchant, such as a server computer executing one or more software applications.

[0070] As used herein, a “point-of-sale (POS) device” may refer to one or more devices, which may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include peripheral devices, card readers, scanning devices (e.g., code scanners), Bluetooth® communication receivers, near-field communication (NFC) receivers, radio frequency identification (RFID) receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, and / or the like. As used herein, a “point- of-sale (POS) system” may refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system may include one or more POS devices and / or other like devices that may be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers programmed or configured to process online payment transactions through webpages, mobile applications, and / or the like.

[0071] As used herein, the terms “client” and “client device” may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a “client device” may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, one or more computing devices used by a payment device provider system, and / or the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, and / or the like), PDAs, and / or the like. Moreover, a “client” may also refer to an entity (e.g., a merchant, an acquirer, and / or the like) that owns, utilizes, and / or operates a client device for initiating transactions (e.g., for initiating transactions with a transaction service provider).

[0072] As used herein, the term “payment device” may refer to a payment card (e.g., a credit or debit card), a gift card, a smartcard, smart media, a payroll card, ahealthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, an RFID transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, a name of the account holder, and / or the like).

[0073] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, POS devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.” Reference to “a server” or “a processor,” as used herein, may refer to a previously recited server and / or processor that is recited as performing a previous step or function, a different server and / or processor, and / or a combination of servers and / or processors. For example, as used in the specification and the claims, a first server and / or a first processor that is recited as performing a first step or function may refer to the same or different server and / or a processor recited as performing a second step or function.

[0074] As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa® or any other entity that processes transactions. The term “transaction processing system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0075] The systems, methods, and computer program products described herein provide numerous technical advantages in systems that employ classification machinelearning models, such as systems for classifying network activity. Model classification accuracy and reliability is improved by training an unsupervised learning model alongside the supervised learning model. In this manner, the system may be able to not only classify data (e.g., using a trained supervised learning model), but further assess whether such a classification would be reliable (e.g., using a trained unsupervised learning model). For example, the unsupervised learning model may be configured to output an outlier score associated with each classification of a plurality of possible classifications, such that reliability assessments are class-specific. This provides a number of technical advantages, including that if activity data is determined to be unreliable for use in classification (e.g., the activity data is an outlier to expected data patterns), then classification may be performed only when the activity data is sufficiently reliable for classification, thereby reducing computing resources (e.g., memory, bandwidth, processing capacity, etc.) throughout the classification pipeline (e.g., at the time of running the model, downstream actions based on the classification, etc.). Moreover, by using an unsupervised learning model to assess classification reliability, no ground-truth labels are needed, which provides for real-time monitoring that is resilient against false positives and false negatives.

[0076] Additionally, by making reliability determinations specific to classifications (e.g., each outlier score based on the trained unsupervised learning model is associated with a classification of the plurality of classifications), then system performance is significantly improved by overcoming data imbalance, data bias, and the false positives / negatives that may result from overly generalized reliability assessments. In some non-limiting embodiments or aspects, system performance is also improved via an iterative, self-improving model training loop. Described systems and methods provide for repeating a performance monitoring loop over a plurality of time periods. In each loop, the described systems and methods provide for receiving new activity data, determining new outlier scores, comparing the new outlier scores to a threshold, and determining whether a prediction of a classification for the new activity data would be unreliable. In response to a number of predictions that are determined to be unreliable satisfying a threshold number, one or more models may be automatically updated by retraining the models using new training data. In this manner, though data patterns for classifications may shift over time, the models may be automatically updated to account for those shifts to maintain optimal performance. In some non-limiting embodiments or aspects, described systems and methods furtherprovide for intelligent processes of initializing thresholds that are used for the outlier comparison, so that the described systems and methods are optimized to use maximum viable outlier score thresholds from the start. These optimizations likewise save on computing resources (e.g., memory, bandwidth, processing capacity), throughout the classification pipeline (e.g., at the time of running the model, downstream actions based on the classification, etc.).

[0077] Referring now to FIG. 1 , FIG. 1 is a diagram of an example environment 100 in which devices, systems, and / or methods, described herein, may be implemented. As shown in FIG. 1 , environment 100 may include modeling system 102, database 104, computing device 106, and communication network 108. Modeling system 102, database 104, and computing device 106 may interconnect (e.g., establish a connection to communicate) via wired connections, wireless connections, or a combination of wired and wireless connections. In some non-limiting embodiments or aspects, environment 100 may further include a natural language processing system, an advertising system, a fraud detection system, a transaction processing system, a merchant system, a point-of-sale (POS) device, an acquirer system, an issuer system, and / or a payment device.

[0078] Modeling system 102 may include one or more computing devices configured to communicate with database 104 and / or computing device 106 at least partly over communication network 108. Modeling system 102 may be configured to receive data associated with network activity (e.g., transaction data, network access request data, communication data, etc.), which may be used to train one or more machine learning models (e.g., supervised learning models, unsupervised learning models, etc.). Modeling system 102 may also be configured to receive new data associated with post-training network activity (e.g., transaction data in a subsequent time period, network access request data in a subsequent time period, etc.), which may be input to one or more trained machine learning models (e.g., trained supervised learning models) to determine one or more classifications for the new data. Modeling system 102 may further be configured to use trained machine learning models (e.g., trained unsupervised learning models) to determine the reliability of classifications produced based on the new data. Modeling system 102 may include and / or be in communication with database 104. Modeling system 102 may be associated with, or included in a same system as, a natural language processing system, a fraud detection system, an advertising system, and / or a transaction processing system.

[0079] Database 104 may include one or more computing devices configured to communicate with modeling system 102 and / or computing device 106 at least partly over communication network 108. Database 104 may be configured to store data associated with network activity (e.g., transaction data, network access request data, communication data, etc.) in one or more non-transitory computer readable storage media. Database 104 may communicate with and / or be included in modeling system 102.

[0080] Computing device 106 may include one or more processors that are configured to communicate with modeling system 102 and / or database 104 at least partly over communication network 108. Computing device 106 may be associated with a user and may include at least one user interface for transmitting data to and receiving data from modeling system 102 and / or database 104. For example, computing device 106 may show, on a display of computing device 106, one or more outputs (e.g., classifications, outlier scores, etc.) of trained machine learning models (e.g., trained supervised learning models, trained unsupervised learning models) executed by modeling system 102. By way of further example, one or more inputs (e.g., selections of data, data files, data fetch queries, etc.) for trained machine learning models may be determined or received by modeling system 102 via a user interface of computing device 106. In some non-limiting embodiments or aspects, computing device 106 may be a mobile device.

[0081] Communication network 108 may include one or more wired and / or wireless networks over which the systems and devices of environment 100 may communicate. For example, communication network 108 may include a cellular network (e.g., a longterm evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0082] The number and arrangement of devices and networks shown in FIG. 1 are provided as an example. There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks than those shown in FIG. 1. Furthermore, two or moredevices shown in FIG. 1 may be implemented within a single device, or a single device shown in FIG. 1 may be implemented as multiple, distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0083] In some non-limiting embodiments or aspects, modeling system 102 may be programmed or configured to perform one or more steps of a method for classifying network activity based on classification-specific data patterns. For example, modeling system 102 may receive training data in a first time period (e.g., day, week, month, quarter-year, year, etc.) associated with historic (e.g., occurring prior to and / or up to the moment of receipt) network activity. Modeling system 102 may be associated with, included in a same system as, and / or include, a fraud detection system, a network management system, a communication monitoring system, and / or the like. Training data may include one or more data records used for training one or more machine learning models. The network may include, but is not limited to, one or more communicatively connected computing devices, such as an electronic payment processing network (e.g., including one or more payment devices, POS devices, merchant systems, acquirer systems, issuer systems, payment gateways, and / or transaction processing systems), a data processing network (e.g., including a computational cluster and a plurality of distributed computing devices associated with one or more domains), a communication network (e.g., including a plurality of computing devices that transmit and receive messages), and / or the like. Activity in the network may include, but is not limited to, transactions (e.g., person-to-person (P2P) transactions, person-to-merchant (P2M) transactions, etc.), network access requests (e.g., requests to access one or more computing resources in the network, such as to read, modify, and / or delete data), communications (e.g., messages including one or more data packets), and / or the like. The training data that is received may be a subset of a greater dataset, and modeling system 102 may subdivide the greater dataset into a training data set, a validation data set, and / or a testing data set. The training data may be stored in database 104 and retrieved by modeling system 102.

[0084] In some non-limiting embodiments or aspects, modeling system 102 may train a supervised learning model (e.g., a machine learning model that learns from labeled training data to make predictions or decisions about unseen or future data,such as a classification machine learning model) based on the training data that is received. For example, modeling system 102 may train the supervised learning model to produce a trained supervised learning model that is configured to output one or more classifications based on input of data associated with network activity. Modeling system 102 may train the supervised learning model by identifying features of the training data, initializing one or more hyperparameters (e.g., weights) of the supervised learning model, feeding the training data into the supervised learning model, and / or iteratively adjusting the parameters of the supervised learning model (e.g., using gradient descent, etc.) to minimize differences (e.g., loss) between predicted outputs (e.g., classifications) and ground truth labels (e.g., known classifications). The supervised learning model may include, but is not limited to, a logistic regression model, a support vector machine (SVM) model, a decision tree model (e.g., a gradient boosting decision tree model), a random forest classifier model, a Naive Bayes classifier model, a k-nearest neighbors (KNN) model, a neural network model, an ensemble model (e.g., a combination of supervised learning models), and / or the like.

[0085] In some non-limiting embodiments or aspects, modeling system 102 may train an unsupervised learning model (e.g., a machine learning model that learns patterns, relationships, structures, and / or the like in unlabeled data without explicit guidance or supervision from predefined labels or target variables) based on the training data that is received. For example, modeling system 102 may train the unsupervised learning model to produce a trained unsupervised learning model that is configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity. Modeling system 102 may train the unsupervised learning model by preparing the training data (e.g., preprocessing, normalizing, filtering, etc.), initializing one or more hyperparameters (e.g., weights) of the unsupervised learning model, fitting the unsupervised learning model to the training data, and / or iteratively adjusting the hyperparameters to improve the quality of the discovered patterns or structures. The unsupervised learning model may include, but is not limited to, an autoencoder model, an isolation forest model, a clustering model, such as a K-means clustering model, a hierarchical clustering model, a density-based spatial clustering of applications with noise (DBSCAN) model, and / or the like. In some non-limiting embodiments or aspects, the unsupervised learning model may include a machine learning modelassociated with (and trained specifically for) each classification of the plurality of classifications.

[0086] In some non-limiting embodiments or aspects, the outlier score may include, but is not limited to, a value, category, or other output indicative of a likelihood of a data point belonging to a given classification, based on whether the data point is associated with a detected pattern of historic data within the given classification. The trained unsupervised learning model may output an outlier score for an input data point for each classification of the plurality of classifications that the supervised learning model is trained to classify data in. For example, the outlier score may be a value between 0 and 1 , where 0 is associated with no likelihood that the data point is an outlier to the expected pattern of a given classification (e.g., the data point is certainly associated a pattern of the classification), and 1 is associated with certainty that the data point is an outlier to the expected pattern of a given classification (e.g., no likelihood that the data point is associated with a pattern of the classification). By way of further example, if the supervised learning model is trained to classify data belonging to one of five classifications, the unsupervised learning model may be trained to output five outlier scores for an input data point, including one outlier score for each classification.

[0087] In some non-limiting embodiments or aspects, modeling system 102 may be configured to employ the trained supervised learning model and trained unsupervised learning model in a process to classify new data and determine whether such classifications of the new data are reliable. For example, modeling system 102 may receive activity data in a second time period (e.g., day, week, month, quarteryear, year, etc.) subsequent (e.g., including entirely or at least partly after) the first time period. The activity data that is received may be associated with network activity. For example, the activity data may include, but is not limited to, data of one or more transactions, data of one or more network access requests, data of one or more communications, and / or the like.

[0088] In some non-limiting embodiments or aspects, the activity data may be transmitted in real-time (e.g., simultaneously upon receipt, immediately after processing, and / or the like) to modeling system 102 for classification. For example, a transaction processing system may receive a transaction request and transmit at least a portion of transaction data of the transaction request to modeling system 102 for classification. By way of further example, a network management system may receivea network access request and transmit at least a portion of data of the network access request to modeling system 102 for classification. By way of further example, a communication monitoring system may identify a communication between computing devices and transmit at least a portion of data of the communication to modeling system 102 for classification. Real-time transmission of activity data for processing by modeling system 102 (e.g., simultaneous or substantially simultaneous inputting to one or more machine learning models) provides the additional technical benefit of reduced overall computational time and quicker system reaction time for responsive actions (e.g., remediative actions) based on classifications.

[0089] In some non-limiting embodiments or aspects, the activity data may be first stored in a storage medium (e.g., database 104) and transmitted to modeling system 102 for classification (e.g., in batch processing). For example, a transaction processing system may process one or more transactions and store at least a portion of transaction data of the one or more transactions in database 104, and modeling system 102 may thereafter receive transaction data from database 104. By way of further example, a network management system may process one or more network access requests and store at least a portion of data of the network access requests in database 104, and modeling system 102 may thereafter receive the data of the one or more network access requests from database 104. By way of further example, a communication monitoring system may identify one or more communications between computing devices and store at least a portion of data of the one or more communications in database 104, and modeling system 102 may thereafter receive the data of the one or more communications from database 104. Delayed and / or batch processing of activity data by modeling system 102 (e.g., temporally separated inputting to one or more machine learning models) provides the additional technical benefit of being able to use computing resources to generate classifications during times of less network / system demand.

[0090] In some non-limiting embodiments or aspects, modeling system 102 may determine a plurality of outlier scores for the activity data. For example, modeling system 102 may generate one or more outputs from the trained unsupervised learning model by inputting one or more data points of the activity data into the trained unsupervised learning model. The outlier scores may be based on the one or more outputs of the trained unsupervised learning model. Each outlier score of the plurality of outlier scores may be associated with a classification of the plurality ofclassifications. For example, modeling system 102 may input a data point of the activity data to the unsupervised learning model and determine a first outlier score of 0.7 for a first classification, a second outlier score of 0.5 for a second classification, a third outlier score of 0.3 for a third classification, and a fourth outlier score of 0.9 for a fourth classification. To further illustrate, for a new data point, modeling system 102 may feed the data point’s features into the classification-dependent unsupervised learning model and obtain an outlier score for each of K classifications (e.g., [s?, S2, S ]). The outlier scores represent how similar the new pattern is compared to the common pattern of each classification that the unsupervised learning model saw during the training process.

[0091] In some non-limiting embodiments or aspects, modeling system 102 may compare each outlier score of the plurality of outlier scores to at least one threshold. The threshold may be of the same data type as the outlier score and may be a value selected within a range of values. For example, for numerical outlier scores (e.g., values between 0 and 1 ), a threshold may be a numerical value (e.g., a value between 0 and 1 ). By way of further example, for categorical outlier scores (e.g., not an outlier, unlikely outlier, likely outlier, outlier), a threshold may be a categorical value (e.g., likely outlier, outlier). The threshold may be predetermined or dynamic. Predetermined thresholds may be set in advance based on user input, statistical analysis, and / or the like. Dynamic thresholds may be periodically adjusted based on user input, statistical analysis, and / or the like. In response to one or more outlier scores (e.g., one outlier score, some outlier scores, all outlier scores) of the plurality of outlier scores satisfying the at least one threshold, modeling system 102 may generate a classification of the activity data based on the trained supervised learning model. By comparing outlier scores to the at least one threshold, modeling system 102 may assess whether classifications of the activity data are likely to be reliable (e.g., one, some, many, or all outlier scores satisfy the threshold), or are likely to be unreliable (e.g., one, some, many, or all outlier scores do not satisfy the threshold). It will be appreciated that by determining whether classifications would be reliable as a precondition step for generating one or more classifications, computer resources (e.g., memory, bandwidth, processing capacity, etc.) would be saved by avoiding generating low-reliability classifications. Moreover, responsive actions based on unreliable classifications may be avoided, providing further technical efficiencies.

[0092] In some non-limiting embodiments or aspects, modeling system 102 may trigger one or more actions based on the generated classification of the trained supervised learning model. For example, the classification of the activity data may be associated with classifying a transaction (e.g., a transaction indicative of impending credit default, a churn prediction based on a transaction, etc.), a network access request (e.g., domain association, permissions level, etc.), a communication (e.g., blacklist, whitelist, spam, etc.), and / or the like. By way of further example, the classification of the activity data may be associated with a type of anomalous (e.g., atypical) network activity (e.g., anomalous transaction, anomalous network access request, anomalous communication), such as where the activity is associated with an atypical and / or unauthorized network event. In response to the classification being associated with a type of anomalous network activity, modeling system 102 may perform, or cause to be performed (e.g., by triggering another system), at least one remediative action (e.g., an action configured to remedy and / or mitigate the effects of the network activity). The at least one remediative action may include, but is not limited to, transmitting at least one alert to a computing device of a user, disabling at least one user account associated with the activity data, changing permissions of at least one user account associated with the activity data, and / or the like.

[0093] In some non-limiting embodiments or aspects, where the at least one remediative action includes transmitting an alert, the at least one remediative action may include notifying an issuer system of an anomalous transaction (e.g., a fraudulent transaction) associated with a payment device issued by the issuer of the issuer system, notifying a payment device holder of an anomalous transaction associated with the holder’s payment device, notifying a system admin of an anomalous network access request of a computing resource in the network, notifying a system user of an anomalous network access request associated with the system user’s account, notifying a spam detection system of an anomalous communication (e.g., a spam message) transmitted in the network, notifying a network user of an anomalous communication transmitted from the user’s account, and / or the like. In some nonlimiting embodiments or aspects, where the at least one remediative action includes disabling at least one user account, the remediative action may include disabling a transaction account associated with a payment device associated with an anomalous transaction, disabling a network user account associated with an anomalous network access request, disabling a telecommunication account associated with an anomalouscommunication, and / or the like. In some non-limiting embodiments or aspects, where the at least one remediative action includes changing permissions of at least one user account, the at least one remediative action may include setting a transaction amount limit for a transaction account associated with an anomalous transaction, triggering additional authentication requirements for the account, restricting read / write access to one or more computing devices in the network, restricting the type or number of communications that can be sent in the network, and / or the like. In this manner, computationally wasteful anomalous activity can be mitigated and / or abated entirely through accurate remediative action.

[0094] In some non-limiting embodiments or aspects, the at least one threshold used for determining reliability may include a plurality of thresholds (e.g., a threshold corresponding to each outlier score for each classification of the plurality of classifications). Modeling system 102 may be configured to execute a process for optimally determining the plurality of thresholds based on the training data, providing further technical efficiencies. For example, modeling system 102 may determine a plurality of candidate thresholds. A candidate threshold may be a potential value (e.g., 0.1 , 0.2, 0.3, 0.4, etc.) that can be used as a threshold in the plurality of thresholds. The plurality of candidate thresholds may include a number of different thresholds that are possible to be used as thresholds (e.g., {0.7, 0.7, 0.7}, {0.3, 0.5, 0.7}, {0.5, 0.5, 0.7}, and / or the like). It is from the plurality of candidate thresholds that modeling system 102 may determine (i) which values of candidate thresholds to use, and (ii) which candidate thresholds should be assigned to specific classifications.

[0095] In some non-limiting embodiments or aspects, modeling system 102 may determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications. For example, each combination may include a plurality of pairings, where each pairing in a combination includes a candidate threshold that is associated with (e.g., paired with) a classification. By way of further example, there may be M candidate thresholds (e.g., where M=10 if the set of candidate thresholds is {0.1 , 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}) and K classifications (e.g., where K=3 if the set of classifications is {first classification, second classification, third classification}). The number of combinations may, therefore, be MK. To illustrate, a first combination of candidate thresholds may be {0.1 , 0.1 , 0.1}, corresponding to the first, second, and third classification respectively. A second combination may be {0.2, 0.1 , 0.1}. A third combination may be {0.3, 0.1 , 0.1}. AnMth combination may be {1 .0, 0.1 , 0.1 }. An M+1th combination may be {0.1 , 0.2, 0.1 }. An MKth combination may be {1.0, 1.0, 1.0}. Modeling system 102 may use a grid search algorithm to traverse the K classes and obtain all MKcandidate threshold combinations (see, e.g., FIG. 1 1 ).

[0096] To determine the optimal combination of candidate thresholds, modeling system 102 may then generate a plurality of training scores based on the trained unsupervised learning model and at least a portion of the training data. Additionally or alternatively, a separate data set may be used for generating the plurality of training scores. The training scores may be a plurality of outlier scores for a plurality of data points of the training data, which are specifically generated for determining an optimal combination of candidate thresholds. For example, modeling system 102 may input a plurality of data points of the training data to the trained unsupervised learning model, receive a plurality of outputs from the trained unsupervised learning model based on the plurality of data points that were input, and generate each training score e.g., outlier score used for training) of the plurality of training scores based on an output of the plurality of outputs. Modeling system 102 may then determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores. Each performance metric may be associated with a pairing of the plurality of combinations. For example, modeling system 102 may determine a number (e.g., a proportion) of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric. Modeling system 102 may base the performance metric on the number. In this manner, modeling system 102 may generate MKperformance metrics.

[0097] Modeling system 102 may then determine a plurality of maximum thresholds for outlier scores based on the plurality of performance metrics. For example, modeling system 102 may determine a plurality of maximum thresholds for outlier scores, for which the plurality of performance metrics satisfy a predetermined threshold metric score. The predetermined threshold metric may include, but is not limited to, an area under the curve (AUC) score, such as an area under a precision recall curve (PR-AUC) score. By way of further example, the plurality of performance metrics may include a user-selected performance metric, which may be defined based on input by a user (e.g., via computing device 106). Modeling system 102 may compare each performance metric (e.g., including an AUC score thereof, a user- selected performance metric thereof, etc.) of the plurality of performance metrics to athreshold score (e.g., 0.7, for an AUC score). Modeling system 102 may then determine a subset of performance metrics of the plurality of performance metrics that include performance metrics (e.g., user-selected performance metrics, AUC scores, etc.) that satisfy the threshold score, but do not necessarily have maximum scores. Modeling system 102 may then determine the plurality of maximum thresholds for outlier scores based on the subset of performance metrics that satisfy the threshold score. Modeling system 102 may further determine the plurality of thresholds to use for the reliability assessment of classifications of activity data based on the plurality of maximum thresholds (e.g., wherein the plurality of thresholds are the plurality of maximum thresholds). See, e.g., FIG. 11 for further detailed description of a method for automatically generating the plurality of thresholds.

[0098] In some non-limiting embodiments or aspects, modeling system 102 may monitor the performance of trained unsupervised learning model to determine when patterns in data have shifted to the point where the unsupervised learning model must be retrained to learning the new data patterns. The monitoring process may include an iterative loop, where certain steps are repeated over a plurality of time periods (e.g., days, weeks, months, etc.). In each time period, modeling system 102 may receive new activity data associated with new network activity. Modeling system 102 may determine a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, wherein each new outlier score is associated with a classification. For example, modeling system 102 may input at least one data point of the new activity data into the trained unsupervised learning model and determine the plurality of new outlier scores based on the output of the trained unsupervised learning model. Modeling system 102 may then compare each new outlier score of the plurality of new outlier scores to the at least one threshold. If a sufficient number of outlier scores satisfy the at least one threshold (e.g., one, some, many, most, all, etc.), modeling system 102 may loop again in a new time period and continue to use the same trained unsupervised learning model. If an insufficient number of outlier scores satisfy the at least one threshold (e.g., zero, most, many, some, one, etc.), then modeling system 102 may determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable. If this determination of unreliability is made a number of times (e.g., once or more for one or more time periods), modeling system 102 may trigger an update process. Modeling system 102 may compare the number of determinations of unreliability to athreshold (e.g., a predetermined number, a predetermined proportion (e.g., 75% reliable), etc.) to trigger the update process.

[0099] In response to the number of predictions over the plurality of time periods determined to be unreliable satisfying the threshold number, modeling system 102 may update the trained supervised learning model and / or the trained unsupervised learning model. Modeling system 102 may update either or both models by receiving new training data based on network activity over the plurality of time periods that drop in reliability was detected, and retraining the supervised learning model and / or unsupervised learning model based on the new training data.

[0100] Referring now to FIG. 2, FIG. 2 is a diagram of example components of a device 200 according to some non-limiting embodiments or aspects. Device 200 may correspond to one or more devices of modeling system 102, database 104, computing device 106, and / or communication network 108 as shown in FIG. 1. In some nonlimiting embodiments or aspects, such systems or devices may include at least one device 200 and / or at least one component of device 200.

[0101] As shown in FIG. 2, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212, and communication interface 214. Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0102] Storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.) and / or another type of computer-readable medium.

[0103] Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally, or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.).

[0104] Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0105] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and / or storage component 208. A computer-readable medium (e.g., a non-transitory computer-readable medium) is defined herein as a non-transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices.

[0106] Software instructions may be read into memory 206 and / or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments or aspects described herein are not limited to any specific combination of hardware circuitry and software.

[0107] The number and arrangement of components shown in FIG. 2 are provided as an example. In some non-limiting embodiments, device 200 may include additionalcomponents, fewer components, different components, or differently arranged components than those shown in FIG. 2. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0108] Referring now to FIG. 3, FIG. 3 is a flowchart of a non-limiting embodiment or aspect of a process 300 for classifying network activity based on classificationspecific data patterns, according to some non-limiting embodiments or aspects. The steps shown in FIG. 3 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some nonlimiting embodiments or aspects. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, and / or the like) by modeling system 102. In some non-limiting embodiments or aspects, one or more of the steps of process 300 may be performed (e.g., completely, partially, and / or the like) by another system, another device, another group of systems, or another group of devices, separate from or including modeling system 102.

[0109] As shown in FIG. 3, at step 302, process 300 may include receiving training data in a first time period. For example, modeling system 102 may receive training data in a first time period associated with historic network activity. By way of further example, modeling system 102 may receive the training data in real-time, delayed, and / or batched communications, and modeling system 102 may receive the training data from database 104.

[0110] As shown in FIG. 3, at step 304, process 300 may include training a supervised learning model based on the training data. For example, modeling system 102 may train a supervised learning model based on the training data to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity.

[0111] As shown in FIG. 3, at step 306, process 300 may include training an unsupervised learning model based on the training data. For example, modeling system 102 may train an unsupervised learning model based on the training data to produce a trained unsupervised learning model configured to output an outlier score (e.g., a score indicative of how similar or dissimilar a data point is from a known pattern) associated with each classification of the plurality of classifications given an input data point associated with network activity.

[0112] As shown in FIG. 3, at step 308, process 300 may include receiving activity data in a second time period. For example, modeling system 102 may receive activity data in a second time period subsequent the first time period associated with network activity. By way of further example, modeling system 102 may receive the activity data in real time, delayed, and / or batched communications, and modeling system 102 may receive the activity data from database 104.

[0113] As shown in FIG. 3, at step 310, process 300 may include determining a plurality of outlier scores for the activity data. For example, modeling system 102 may determine a plurality of outlier scores for the activity data, wherein each outlier score of the plurality of outlier scores is associated with (e.g., corresponds to) a classification of the plurality of classifications.

[0114] As shown in FIG. 3, at sub-step 312, process 300 (and step 310) may include inputting the activity data to the trained unsupervised learning model. For example, modeling system 102 may, when determining the plurality of outlier scores, input the activity data to the trained unsupervised learning model.

[0115] As shown in FIG. 3, at sub-step 314, process 300 (and step 310) may include determining the plurality of outlier scores based on output of the trained unsupervised learning model. For example, modeling system 102 may, when determining the plurality of outlier scores, determine the plurality of outlier scores based on at least one output of the trained unsupervised learning model.

[0116] As shown in FIG. 3, at step 316, process 300 may include comparing the plurality of outlier scores to a threshold. For example, modeling system 102 may compare each outlier score of the plurality of outlier scores to at least one threshold.

[0117] In some non-limiting embodiments or aspects, the at least one threshold may include a plurality of thresholds. In such a scenario, modeling system 102 may further execute a process for determining the plurality of thresholds. For example, modeling system 102 may determine a plurality of candidate thresholds. Modeling system 102 may also determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations includes a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination of the plurality of combinations includes a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications. Modeling system 102 may further generate a plurality of training scores based on the trained unsupervised learningmodel and the training data. Modeling system 102 may further determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores. Modeling system 102 may further determine a plurality of maximum thresholds based on the plurality of performance metrics. Modeling system 102 may further determine the plurality of thresholds based on the plurality of maximum thresholds.

[0118] In some non-limiting embodiments or aspects, when generating the plurality of training scores based on the trained unsupervised learning model and the training data, modeling system 102 may input a plurality of data points of the training data to the trained unsupervised learning model, receive a plurality of outputs from the trained unsupervised learning model based on the plurality of data points, and generate each training score of the plurality of training scores based on an output of the plurality of outputs.

[0119] In some non-limiting embodiments or aspects, each performance metric of the plurality of performance metrics may be associated with a combination of the plurality of combinations. When determining the plurality of performance metrics, modeling system 102 may, for each performance metric of the plurality of performance metrics, determine a number of the plurality of training scores that satisfy the plurality of candidate thresholds associated with the plurality of classifications for the combination associated with the performance metric. The performance metric may be based on the number. The plurality of performance metrics may include a user- selected performance metric. When determining the plurality of maximum thresholds based on the plurality of performance metrics, modeling system 102 may compare each performance metric of the plurality of performance metrics to a threshold score, determine a subset of the performance metrics of the plurality of performance metrics that include performance metrics that satisfy the threshold score, and determine the plurality of maximum thresholds based on the subset of performance metrics.

[0120] As shown in FIG. 3, at step 318, process 300 may include generating a classification of the activity data based on the trained supervised learning model. For example, modeling system 102 may, in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model. Step 318 may be performed in addition to or alternatively from step 320.

[0121] In some non-limiting embodiments or aspects, the classification of the activity data may be associated with a type of anomalous network activity. Modeling system 102 may, in response to determining a classification of the activity data that is associated with a type of anomalous network activity, perform one or more remediative actions based on the classification. The one or more remediative actions may include, but are not limited to, transmitting at least one alert to computing device 106 of a user, disabling at least one user account associated with the activity data, changing permissions of at least one user account associated with the activity data, and / or the like.

[0122] As shown in FIG. 3, at step 320, process 300 may include determining that the activity data is unreliable for classification. For example, modeling system 102 may, in response to one or more outlier scores of the plurality of outlier scores not satisfying the at least one threshold, determining that the activity data is unreliable for classification based on the trained supervised learning model.

[0123] In some non-limiting embodiments or aspects, at least part of process 300 may be performed cyclically to monitor the ongoing reliability and accuracy of the machine learning models. For example, modeling system 102 may perform a set of steps over a plurality of time periods. By way of further example, modeling system 102 may, in each loop at step 308, receive new activity data in a new time period of the plurality of time periods, the new activity data associated with new network activity. Modeling system 102 may, in each loop at step 310 (including sub-steps 312 and 314), determine a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications. Modeling system 102 may, in each loop at step 316, compare each new outlier score of the plurality of outlier scores to the at least one threshold. Modeling system 102 may, in each loop, perform either step 318 or step 320 based on the comparison of each new outlier score of the plurality of new outlier scores to the at least one threshold. For example, modeling system 102 may, in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable. While looping, modeling system 102 may further compare a number of predictions determined to be unreliable to a threshold number (e.g., of a tolerable number of determinations of unreliability).Modeling system 102 may, in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying the threshold number, update the trained supervised learning model and / or the trained unsupervised learning model. When updating the trained supervised learning model and / or trained unsupervised learning model, modeling system 102 may receive new training data based on network activity over the plurality of time periods and train the supervised learning model and / or the unsupervised learning model based on the new training data. Such training may include training entirely anew without the learning and inferences of prior training, or such training may be additive upon prior learning and inferences.

[0124] Referring now to FIG. 4, FIG. 4 is a plurality of bar graphs illustrating changes in data patterns over time, according to some non-limiting embodiments or aspects. FIG. 4 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. As shown, FIG. 4 demonstrates the importance of monitoring machine learning model performance over time to detect changes in reliability due to shifting data patterns. Depicted is a first bar graph 400A illustrating a first data pattern in a first time period, a second bar graph 400B illustrating a second data pattern in a second time period (e.g., a period six months after the first time period), and a third bar graph 400C illustrating third data pattern in a third time period (e.g., a period six months after the second time period and twelve months after the first time period). All three illustrative bar graphs 400A, 400B, 400C are for the same data parameters, namely, customer age distribution of a marketed product (e.g., y- axis showing amount of customers, x-axis showing age of customers). In the first time period (shown in first bar graph 400A), the customer age distribution of a marketed product tends toward the lower age ranges (e.g., ages [20,40], with a strong peak at age 30). Now, consider where a new product feature is implemented after the first time period, which may cause the customer age distribution to change. In the second time period (shown in second bar graph 400B), the customer age distribution of the marketed product has shifted wider (e.g., ages [20,50], with a less strong peak at age 30). Now, consider where a marketing strategy changes for the product after the second time period, which causes the customer age distribution to change again. In the third time period (shown in third bar graph 400C), the customer age distribution of the marketed product has shifted wider (e.g., ages [20,60], with a less strong peak along ages [25, 35]). As illustrated, a supervised learning model (e.g., a classification model) trained on the data of the first time period (bar graph 400A) may not necessarilybe reliable for making classifications in the second time period (bar graph 400B) or the third time period (bar graph 400C).

[0125] Referring now to FIG. 5, FIG. 5 is a pair of scatterplots illustrating a detected unreliable classification of data, according to some non-limiting embodiments or aspects. FIG. 5 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. As shown, FIG. 5 depicts a first scatterplot 500A from a training phase for an unsupervised learning model, and a second scatterplot 500B from a testing phase for an unsupervised learning model. In the training phase, as shown in first scatterplot 500A, modeling system 102 has trained at least an unsupervised learning model to detect patterns of classifications. A data pattern of a first class is shown by solid points in first scatterplot 500A, and a data pattern of a second class is shown by hollow points in first scatterplot 500A. In a subsequent testing phase, as shown in second scatterplot 500B, a data point 502 (solid triangle) is provided that is outside of the training sample space (denoted by dashed lines defining an area in each scatterplot 500A, 500B). Unsupervised learning model would detect that data point 502 is an outlier to either the data pattern of class 1 or the data pattern of class 2. A classification based on data point 502 would be unreliable based on the detected outlier status of data point 502 (e.g., which may be determined based on an outlier score).

[0126] Referring now to FIG. 6, FIG. 6 is a flowchart illustrating a process 600 for data classification based on classification-specific data patterns, according to some non-limiting embodiments or aspects of the present disclosure. FIG. 6 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. The process 600 includes receiving training data 602 and training a supervised learning model 604 and an unsupervised learning model 606. The supervised learning model 604 is trained to produce a first output 610 associated with a classification (e.g., a category, a probability of belonging to a group, etc.). The unsupervised learning model 606 is trained to produce a second output 612 associated with each classification (e.g., an outlier score representative of a likelihood of a data point belonging to a data pattern of the classification, such as where a high outlier score indicates a likelihood of not belonging to a classification).

[0127] After training, new data 608 is received and used as input to the trained supervised learning model 604 and unsupervised learning model 606. In some nonlimiting embodiments or aspects, new data 608 may not be input into trainedsupervised learning model 604 until the results of the output from unsupervised learning model 606 are evaluated. New data 608 may include, but is not limited to, activity data associated with network activity. By inputting new data 608 into supervised learning model 604, a first output 610 is produced, and by inputting new data 608 into unsupervised learning model 606, a second output 612 is produced. Both first output 610 and second output 612 are used to make a final decision 614. The final decision 614 may be based on the decision table shown below.Table 1As shown above, if an outlier score (of second output 612) associated with classification x does not satisfy a threshold (e.g., meets or exceeds a tolerable likelihood of outlier), first output 610 of trained supervised learning model 604 may be determined to be unreliable. Conversely, if an outlier score (of second output 612) associated with classification x does satisfy a threshold (e.g., meets or falls below a tolerable likelihood of outlier), first output 610 of trained supervised learning model 604 may be determined to be sufficiently reliable. The decision process reflected in Table 1 may be executed for each classification of the plurality of classifications, since second output 612 is classification-specific. In some non-limiting embodiments or aspects, if second output 612 shows an outlier score for a classification that does not satisfy the threshold, first output 610 may not be generated (e.g., skipped), and the decision process may proceed to evaluate the second output 612 for the next classification.

[0128] With further reference to FIG. 6, first output 610 from supervised learning model 604 may be combined with second output 612 of unsupervised learning model606. Considering N data points with K possible classes to be assigned, the results from both models 604, 606 may be represented by the following table:Table 2As shown in Table 2, the probability Pn,k (where n=1,..., N and k=1,...,K) shows how likely the new point belongs to each class Ck (where k=1,...,K), inferred from the supervised learning model 604. The outlier score OSn,k shows how dissimilar the new point is against each class’s common pattern, inferred from the class-dependent unsupervised learning model 606.

[0129] Referring now to FIG. 7, FIG. 7 is a pair of scatterplots 700A, 700B showing outlier scoring using an unsupervised learning model, according to some non-limiting embodiments or aspects. FIG. 7 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. For a set of feature data (e.g., derived from network activity data), an unsupervised learning model (e.g., an isolation forest model) may quantify how different one point is from the set of data points, using an outlier score. For ease of illustration, FIG. 7 shows a process for a two-class dataset with 200 points while each point has two features, x1 and x2. While a two-dimensional dataset is provided, it will be appreciated that the same techniques may be applied to datasets of higher dimensionality. The first scatterplot 700A shows the dataset without outlier scoring (e.g., before being input to unsupervised learning model). The second scatterplot 700B shows the dataset with group outlier scoring applied (e.g., after being input to supervised learning model). As shown, points toward the center of the data pattern (e.g., point 702) have a lower outlier score (represented visually by darker luminosity-based representation), meaning that such points are likely to belong to the group data pattern. Points away from the center of the data pattern (e.g., point 704) have a higher outlier score (represented visually by lighter luminosity-based representation), meaning that such points are not likely to belong to the group data pattern. While a group outlier score is shown for ease of reference, it will beappreciated that one-dimension (e.g., per-classification) outlier scores may be output by unsupervised learning model for each data point (see, e.g., FIG. 8).

[0130] Referring now to FIG. 8, FIG. 8 is a scatterplot 800 showing classificationbased outlier scoring using an unsupervised learning model, according to some nonlimiting embodiments or aspects. FIG. 8 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. Scatterplot 800 is a graph of a training dataset (representing by circular data points) having two dimensions e.g., classifications, features, etc.). A first classification exhibits a first data pattern (e.g., a cluster) on the left side of scatterplot 800, and a second classification exhibits a second data pattern (e.g., a cluster) on the right side of scatterplot 800. After training, new data points 802, 804 (represented by X-shaped data points) may be input to unsupervised learning model to output outlier scores for new data points 802, 804. The outlier scores produced for each new data point 802, 804 are shown in the below table (e.g., with outlier scores provided in a range between 0 and 1 , with 1 being a high likelihood of outlier).Table 3As shown, new data point 802 has a relatively small outlier score for a second classification (class 2), while new point 804 has a relatively large outlier score for either classification. A supervised learning model’s reliability to classify new data point 804 will be much lower than in classifying new data point 802.

[0131] Referring now to FIG. 9, FIG. 9 is a pair of scatterplots 900A, 900B showing use cases of classification-based outlier scoring in using an unsupervised learning model, according to some non-limiting embodiments or aspects. FIG. 9 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. In each scatterplot, data of a first class are represented by solid, round data points, data of a second class are represented by hollow, round data points, and new data (e.g., test data) are represented by X-shaped data points. As shown in first scatterplot 900A, when data are imbalanced (e.g., certain classes have disproportionately fewer samples than other classes), a global-level outlier scoringmodel may output higher outlier scores for the minority class data. Therefore, the new point of first scatterplot 900A may be assigned an artificially high global outlier score when using a global outlier scoring model, even though the new data point is fairly representative of the minority class 2 and should have a low outlier score for class 2. Similarly, as shown in second scatterplot 900B, model reliability may differ for new data points even if their global outlier scores are relatively the same. For example, the far left new data point of second scatterplot 900B is fairly close to the data pattern of class 1 , which may mean that classification for that point would be more reliable for class 1 . All three new data points, using a global-level outlier score, may have very similar outlier scores, despite their individual data patterns being, in fact, more similar to one class of data or another. It will be appreciated that assessing reliability with classification-specific outlier scoring models increases model sensitivity.

[0132] Referring now to FIGS. 10A and 10B, FIGS. 10A and 10B are graphical representations of the relationship between outlier scores and performance metrics, according to some non-limiting embodiments or aspects. FIGS. 10A and 10B are provided for illustrative purposes only and are not to be taken as limiting on the present disclosure. A supervised learning model’s predictions (e.g., classifications) on data with unseen patterns (e.g., having overall higher outlier scores) may be more random than those with common data patterns. As a result, performance metrics (e.g., PR- AUC) will drop, indicating low model reliability.

[0133] Depicted in FIG. 10A is a first scatterplot 1000A showing historical data of a first class (solid, round data points), historical data of a second class (hollow, round data points), and new data points (X-shaped data points). The new data points of first scatterplot 1000A exhibit a different pattern from the known patterns of either class 1 or class 2, and, therefore, the new data points will have higher overall outlier scores. Higher outlier scores indicate lower classification reliability, and graph 1000B illustrates the associated PR-AUC score for classifying the new data of shown in first scatterplot 1000A. As shown, the PR-AUC score is lower than in FIG. 10B.

[0134] Depicted in FIG. 10B is a second scatterplot 1 100A showing historical data of a first class (solid, round data points), historical data of a second class (hollow, round data points), and new data points (X-shaped data points). The new data points of second scatterplot 1 100A exhibit a similar pattern to the known patterns of either class 1 or class 2 (or both), and, therefore, the new data points will have lower overall outlier scores. Lower outlier scores indicate higher classification reliability, and graph1 100B illustrates the associated PR-AUC score for classifying the new data shown in second scatterplot 1 100A. As shown, the PR-AUC score is higher than in FIG. 10A.

[0135] Referring now to FIG. 1 1 , FIG. 1 1 is a graphical representation of determining outlier score thresholds based on performance metrics, according to some non-limiting embodiments or aspects. FIG. 1 1 is provided for illustrative purposes only and is not to be taken as limiting on the present disclosure. In some non-limiting embodiments or aspects, modeling system 102 may quantify the relationship between model performance and outlier scores, such that thresholds for the outlier scores may be optimally set to achieve an acceptable performance level. In a first step, for each class k, modeling system 102 may create a vector of outlier score threshold values (e.g., candidate thresholds) from 0 to 1 uniformly, e.g., limk = [0, 0.1, ..., 1], In a second step, modeling system 102 may use grid search across the K classes to obtain all score threshold value combinations (e.g., where each combination includes a number of pairings between a candidate threshold and a classification). If there are M candidate thresholds limk for k classes, modeling system 102 may obtain MKcombinations. For example, a first combination may be limi=0.1, lirri2=0.1, ..., lirrik=0.1, a second combination may be limi=0.2, lirri2=0.1, ..., lirrik=0.1, and so on, until combination MK, which may be limi=1.0, lirri2=1.0, ..., lirrik=1.0. It will be appreciated that other sets of candidate thresholds may be used besides the depicted uniform distribution between 0 and 1. In a third step, modeling system 102 may, for every score combination, find the data in the historical dataset whose outlier scores are below the thresholds. In a fourth step, modeling system 102 may, for each subset data obtained from the third step, calculate a performance metric (e.g., PR- AUC) to produce MKperformance metric values corresponding to different outlier score thresholds.

[0136] With further reference to FIG. 1 1 , modeling system 102 may determine an acceptable performance level for a developed machine learning model. The acceptance performance level may be based on user input, past performance, and / or the like. Based on the relationship between outlier score thresholds and performance metric values, modeling system 102 may decide whether the model is reliable on given data, including determining what thresholds to set for the comparison of outlier scores. Shown in FIG. 1 1 is a binary classification dataset with [s1, s2] as the outlier scores for a first and second class, respectively. PR-AUC scores vary by different thresholds of [s1 , s2] , as represented by the luminosity-based representation of the plot points inthe graph (e.g., darker is lower PR-AUC score, lighter is higher PR-AUC score). If modeling system 102 determines that 0.7 is the lowest PR-AUC score that is acceptable for a classification model, new data with outlier scores s1 <0.5 or s2 <0.4 may be considered as “model reliable”. As shown, the area in the top right of the plot (denoted by an inner square) includes lower PR-AUC scores associated with higher outlier scores. The inverse area includes higher PR-AUC scores associated with lower outlier scores. Outlier scores above the thresholds of s1 = 0.5 and s2 = 0.4 are associated with unreliable modeling. Therefore, modeling system 102 may use those values as maximum thresholds for outlier score comparisons that still satisfy the target model performance.

[0137] With reference to the foregoing figures, the term “class” may be used to refer to a classification of a plurality of classifications that may be output by a supervised learning model, which may represent one or more features of data input to the supervised learning model.

[0138] Although the disclosure has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred embodiments or aspects, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect, and one or more steps may be taken in a different order than presented in the present disclosure.

Claims

WHAT IS CLAIMED IS:1 . A computer-implemented method, comprising: receiving, with at least one processor, training data in a first time period associated with historic network activity; training, with at least one processor, a supervised learning model based on the training data, to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity; training, with at least one processor, an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receiving, with at least one processor, activity data in a second time period subsequent the first time period associated with network activity; determining, with at least one processor, a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein determining the plurality of outlier scores comprises: inputting the activity data to the trained unsupervised learning model; and determining the plurality of outlier scores based on at least one output of the trained unsupervised learning model; comparing, with at least one processor, each outlier score of the plurality of outlier scores to at least one threshold; and in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generating, with at least one processor, a classification of the activity data based on the trained supervised learning model.

2. The computer-implemented method of claim 1 , wherein the at least one threshold comprises a plurality of thresholds, the method further comprising: determining, with at least one processor, a plurality of candidate thresholds;determining, with at least one processor, a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generating, with at least one processor, a plurality of training scores based on the trained unsupervised learning model and the training data; determining, with at least one processor, a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determining, with at least one processor, a plurality of maximum thresholds based on the plurality of performance metrics; and determining, with at least one processor, the plurality of thresholds based on the plurality of maximum thresholds.

3. The method of claim 2, wherein generating the plurality of training scores based on the trained unsupervised learning model and the training data comprises: inputting a plurality of data points of the training data to the trained unsupervised learning model; receiving a plurality of outputs from the trained unsupervised learning model based on the plurality of data points; and generating each training score of the plurality of training scores based on an output of the plurality of outputs.

4. The method of claim 2, wherein each performance metric of the plurality of performance metrics is associated with a combination of the plurality of combinations, and wherein determining the plurality of performance metrics comprises, for each performance metric of the plurality of performance metrics: determining a number of the plurality of training scores that satisfy the plurality of candidate thresholds associated with the plurality of classifications for the combination associated with the performance metric, wherein the performance metric is based on the number.

5. The method of claim 4, wherein the plurality of performance metrics comprises a user-selected performance metric; and wherein determining the plurality of maximum thresholds based on the plurality of performance metrics comprises: comparing each performance metric of the plurality of performance metrics to a threshold score; determining a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determining the plurality of maximum thresholds based on the subset of performance metrics.

6. The method of claim 1 , further comprising: repeating over a plurality of time periods: receiving, with at least one processor, new activity data in a new time period of the plurality of time periods associated with new network activity; determining, with at least one processor, a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications; comparing, with at least one processor, each new outlier score of the plurality of new outlier scores to the at least one threshold; and in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determining, with at least one processor, that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; and in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, updating, with at least one processor, the trained supervised learning model.

7. The method of claim 6, wherein updating the trained supervised learning model comprises:receiving, with at least one processor, new training data based on network activity over the plurality of time periods; and training, with at least one processor, the supervised learning model based on the new training data.

8. The method of claim 1 , wherein the classification of the activity data is associated with a type of anomalous network activity, the method further comprising: performing, with at least one processor, at least one remediative action based on the classification of the activity data, wherein the at least one remediative action comprises: transmitting at least one alert to a computing device of a user; disabling at least one user account associated with the activity data; changing permissions of at least one user account associated with the activity data; or any combination thereof.

9. A system comprising at least one processor programmed or configured to: receive training data in a first time period associated with historic network activity; train a supervised learning model based on the training data, to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity; train an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receive activity data in a second time period subsequent the first time period associated with network activity; determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein, when determining the plurality of outlier scores, the at least one processor is programmed or configured to: input the activity data to the trained unsupervised learning model; anddetermine the plurality of outlier scores based on at least one output of the trained unsupervised learning model; compare each outlier score of the plurality of outlier scores to at least one threshold; and in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

10. The system of claim 9, wherein the at least one threshold comprises a plurality of thresholds, and wherein the at least one processor is further programmed or configured to: determine a plurality of candidate thresholds; determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generate a plurality of training scores based on the trained unsupervised learning model and the training data; determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determine a plurality of maximum thresholds based on the plurality of performance metrics; and determine the plurality of thresholds based on the plurality of maximum thresholds.1 1 . The system of claim 10, wherein each performance metric of the plurality of performance metrics is associated with a pairing of each combination of the plurality of combinations, and wherein, when determining the plurality of performance metrics, the at least one processor is programmed or configured to, for each performance metric of the plurality of performance metrics:determine a number of the plurality of training scores that satisfy the candidate threshold associated with the pairing associated with the performance metric, wherein the performance metric is based on the number.

12. The system of claim 1 1 , wherein the plurality of performance metrics comprises a user-selected performance metric; and wherein, when determining the plurality of maximum thresholds based on the plurality of performance metrics, the at least one processor is programmed or configured to: compare each performance metric of the plurality of performance metrics to a threshold score; determine a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determine the plurality of maximum thresholds based on the subset of performance metrics.

13. The system of claim 9, wherein the at least one processor is further programmed or configured to: repeat over a plurality of time periods: receive new activity data in a new time period of the plurality of time periods associated with new network activity; determine a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications; compare each new outlier score of the plurality of new outlier scores to the at least one threshold; and in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; andin response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

14. The system of claim 13, wherein, when updating the trained supervised learning model, the at least one processor is programmed or configured to: receive new training data based on network activity over the plurality of time periods; and train the supervised learning model based on the new training data.

15. A computer program product comprising at least one non- transitory computer-readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: receive training data in a first time period associated with historic network activity; train a supervised learning model based on the training data, to produce a trained supervised learning model configured to output at least one classification of a plurality of classifications associated with network activity; train an unsupervised learning model based on the training data, to produce a trained unsupervised learning model configured to output an outlier score associated with each classification of the plurality of classifications given an input data point associated with network activity; receive activity data in a second time period subsequent the first time period associated with network activity; determine a plurality of outlier scores for the activity data, each outlier score of the plurality of outlier scores being associated with a classification of the plurality of classifications, wherein, when determining the plurality of outlier scores, the at least one processor is programmed or configured to: input the activity data to the trained unsupervised learning model; and determine the plurality of outlier scores based on at least one output of the trained unsupervised learning model;compare each outlier score of the plurality of outlier scores to at least one threshold; and in response to each outlier score of the plurality of outlier scores satisfying the at least one threshold, generate a classification of the activity data based on the trained supervised learning model.

16. The computer program product of claim 15, wherein the at least one threshold comprises a plurality of thresholds, and wherein the one or more instructions further cause the at least one processor to: determine a plurality of candidate thresholds; determine a plurality of combinations based on the plurality of candidate thresholds and the plurality of classifications, wherein each combination of the plurality of combinations comprises a plurality of pairings, and wherein each pairing of the plurality of pairings of each combination comprises a candidate threshold of the plurality of candidate thresholds associated with a classification of the plurality of classifications; generate a plurality of training scores based on the trained unsupervised learning model and the training data; determine a plurality of performance metrics associated with the plurality of combinations based on the plurality of training scores; determine a plurality of maximum thresholds based on the plurality of performance metrics; and determine the plurality of thresholds based on the plurality of maximum thresholds.

17. The computer program product of claim 16, wherein each performance metric of the plurality of performance metrics is associated with a combination of the plurality of combinations, and wherein the one or more instructions that cause the at least one processor to determine the plurality of performance metrics cause the at least one processor to, for each performance metric of the plurality of performance metrics: determine a number of the plurality of training scores that satisfy the plurality of candidate thresholds associated with the plurality of classifications for thecombination associated with the performance metric, wherein the performance metric is based on the number.

18. The computer program product of claim 17, wherein the plurality of performance metrics comprises a user-selected performance metric; and wherein the one or more instructions that cause the at least one processor to determine the plurality of maximum thresholds based on the plurality of performance metrics cause the at least one processor to: compare each performance metric of the plurality of performance metrics to a threshold score; determine a subset of performance metrics of the plurality of performance metrics comprising performance metrics that satisfy the threshold score; and determine the plurality of maximum thresholds based on the subset of performance metrics.

19. The computer program product of claim 15, wherein the one or more instructions further cause the at least one processor to: repeat over a plurality of time periods: receive new activity data in a new time period of the plurality of time periods associated with new network activity; determine a plurality of new outlier scores for the new activity data based on the trained unsupervised learning model, each new outlier score of the plurality of new outlier scores being associated with a classification of the plurality of classifications; compare each new outlier score of the plurality of new outlier scores to the at least one threshold; and in response to at least one new outlier score of the plurality of new outlier scores not satisfying the at least one threshold, determine that a prediction of a classification for the new activity data based on the trained supervised learning model would be unreliable; and in response to a number of predictions over the plurality of time periods determined to be unreliable satisfying a threshold number, update the trained supervised learning model.

20. The computer program product of claim 19, wherein the one or more instructions that cause the at least one processor to update the trained supervised learning model cause the at least one processor to: receive new training data based on network activity over the plurality of time periods; and train the supervised learning model based on the new training data.

Images