Systems, methods, and computer program products for time-based aggregated learning using supervised and unsupervised machine learning models

Abstract

Systems for aggregating learning with machine learning models are provided, including a processor to receive a training dataset of a plurality of data instances, where each data instance includes a time series of data points, add a time delay amount to one or more data instances to provide an augmented training dataset, select a first plurality of supervised machine learning models, select a second plurality of unsupervised machine learning models, train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset, generate an aggregated machine learning model based on outputs of the supervised machine learning models and the unsupervised machine learning models, and generate a runtime output of the aggregated machine learning model based on a runtime input of the aggregated machine learning model. Methods and computer program products are also provided.

Description

Technical Field

[0001] This disclosure generally relates to machine learning, and in some non-limiting embodiments or aspects, to systems, methods, and computer program products for generating machine learning models based on aggregate learning methods using supervised and unsupervised techniques. Background Technology

[0002] Machine learning can be a field of computer science that uses statistical techniques to give computer systems the ability to learn tasks (e.g., progressively improve performance) using data, without explicitly programming the computer system to perform the tasks. In some cases, machine learning models can be developed for datasets that can perform tasks related to said datasets (e.g., tasks associated with prediction).

[0003] In some cases, machine learning models, such as predictive machine learning models, can be used to make predictions related to risk or opportunity based on large amounts of data (e.g., large-scale datasets). Predictive machine learning models can be used to analyze the relationship between a unit's performance and one or more known features of the unit, based on a large dataset associated with that unit. The purpose of a predictive machine learning model can be to assess the probability that similar units will exhibit the same or similar performance as the unit. To generate a predictive machine learning model, the large-scale dataset can be segmented so that the predictive machine learning model can be trained on appropriate data.

[0004] In some cases, multi-class classification (e.g., polynomial classification) can refer to the problem of classifying an instance into one of three or more categories, while classifying an instance into one of two categories can be called binary classification.

[0005] Deep learning (e.g., deep structured learning) can refer to a category of machine learning algorithms based on artificial neural networks with representation learning, where the representation learning can be supervised, semi-supervised, and / or unsupervised. Machine learning architectures used for deep learning can include deep neural networks (DNNs), deep belief networks, deep reinforcement learning, repetitive neural networks, and convolutional neural networks.

[0006] In the field of deep learning, machine learning models typically excel at learning to perform specific tasks on a given large dataset. As datasets grow and change, machine learning models that have learned to perform a specific task on older datasets become obsolete and need to be trained on newer datasets so that the model can perform with the same accuracy or even better on the new data. This change in model performance as datasets change and expand is known as model freshness degradation.

[0007] Developing machine learning models is extremely time-consuming and requires vast amounts of data. If a classification model is built using a specific dataset, its accuracy may degrade over time if it is not updated and trained with new data. This degradation can be even faster when new data patterns emerge due to sudden changes in the market or technology (e.g., technological shifts, domain shifts, market shifts). Such machine learning models will require continuous training and updates with new and emerging data to maintain or improve their accuracy over time.

[0008] In certain environments, such as online deep learning models, deep learning models often face problems such as data latency and model perturbation during training. Unlike offline models, online models are susceptible to real-time data changes and incomplete datasets. If model performance largely depends on the completeness of the data used for training, such real-time data changes can negatively impact model accuracy. For example, training a classification model on image data can receive many similar images as input, images that are barely different. Even a single image with a tiny difference in a heavily weighted feature can perturb the model and negatively affect its training. Furthermore, if the input image data is time-ordered, some images may be fed into the model inconsistent with the sequence of other images, potentially negatively impacting model training.

[0009] Given the described problem, it is evident that there is a need for training systems, methods, and computer program products based on machine learning / deep learning aggregation to generate machine learning models using supervised and unsupervised techniques that are less susceptible to model degradation, data latency, and model perturbations. Summary of the Invention

[0010] Therefore, what is disclosed are systems, methods, and computer program products for generating machine learning models based on aggregation learning methods using supervised and unsupervised techniques.

[0011] According to some non-limiting embodiments or aspects, a method is provided for aggregate learning using machine learning models to prevent model performance degradation. The system includes at least one processor programmed and / or configured to receive a training dataset of multiple data instances, wherein each data instance includes a time series of data points. The at least one processor is also programmed and / or configured to add a time delay amount to one or more of the multiple data instances to provide an augmented training dataset. The at least one processor is also programmed and / or configured to select a first plurality of supervised machine learning models and a second plurality of unsupervised machine learning models. The at least one processor is further programmed and / or configured to train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset. The at least one processor is further programmed and / or configured to generate an aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models. The at least one processor is further programmed and / or configured to generate the runtime output of the aggregated machine learning model based on the runtime input of the aggregated machine learning model.

[0012] In some non-limiting embodiments or aspects, when the time delay amount is added to one or more of the plurality of data instances to provide the augmented training dataset, the at least one processor is further programmed and / or configured to add the time delay amount to a plurality of labeled data instances to provide a plurality of augmented labeled data instances. The at least one processor is further programmed and / or configured to add the time delay amount to a plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances. The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0013] In some non-limiting embodiments or aspects, the at least one processor is further programmed and / or configured to perform actions based on the runtime output of the aggregated machine learning model.

[0014] In some non-limiting embodiments or aspects, when generating the aggregated machine learning model, the at least one processor is programmed and / or configured to generate the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The at least one processor is further programmed and / or configured to train the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0015] In some non-limiting embodiments or aspects, the at least one processor is further programmed and / or configured to add perturbations to one or more additional data instances among the plurality of data instances. The at least one processor is further programmed and / or configured to add the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0016] In some non-limiting embodiments or aspects, when the first plurality of supervised machine learning models are selected, the at least one processor is programmed and / or configured to determine a sensitivity metric to time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset. The at least one processor is further programmed and / or configured to select a subset of the set of supervised machine learning models based on the sensitivity metric to time delay for each supervised machine learning model. In some non-limiting embodiments or aspects, when the second plurality of unsupervised machine learning models are selected, the at least one processor is programmed and / or configured to determine a sensitivity metric to time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset. The at least one processor is further programmed and / or configured to select a subset of the set of unsupervised machine learning models based on the sensitivity metric to time delay for each unsupervised machine learning model.

[0017] In some non-limiting embodiments or aspects, the sensitivity measure of the supervised machine learning model to time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset and relative to the augmented training dataset. In some non-limiting embodiments or aspects, the sensitivity measure of the unsupervised machine learning model to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset.

[0018] According to some non-limiting embodiments or aspects, a computer-implemented method is provided for using aggregate learning with machine learning models to prevent model performance degradation. The method includes receiving a training dataset of multiple data instances, wherein each data instance includes a time series of data points. The method further includes adding a time delay amount to one or more of the multiple data instances to provide an augmented training dataset. The method further includes selecting a first plurality of supervised machine learning models and selecting a second plurality of unsupervised machine learning models. The method further includes training the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset. The method further includes generating an aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The method further includes generating a runtime output of the aggregated machine learning model based on runtime inputs to the aggregated machine learning model.

[0019] In some non-limiting embodiments or aspects, adding the time delay amount to one or more of the plurality of data instances to provide the augmented training dataset includes: adding the time delay amount to a plurality of labeled data instances to provide a plurality of augmented labeled data instances. The method further includes adding the time delay amount to a plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances. The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0020] In some non-limiting embodiments or aspects, the method further includes performing actions based on the runtime output of the aggregated machine learning model.

[0021] In some non-limiting embodiments or aspects, generating the aggregated machine learning model includes: generating the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The method further includes training the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0022] In some non-limiting embodiments or aspects, the method further includes adding a perturbation to one or more additional data instances among the plurality of data instances. The method further includes adding the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0023] In some non-limiting embodiments or aspects, selecting the first plurality of supervised machine learning models includes determining a sensitivity metric to time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset. The method further includes selecting a subset of the set of supervised machine learning models based on the sensitivity metric to time delay for each supervised machine learning model. In some non-limiting embodiments or aspects, selecting the second plurality of unsupervised machine learning models includes determining a sensitivity metric to time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset. The method further includes selecting a subset of the set of unsupervised machine learning models based on the sensitivity metric to time delay for each unsupervised machine learning model.

[0024] In some non-limiting embodiments or aspects, the sensitivity measure of the supervised machine learning model to time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset and relative to the augmented training dataset. In some non-limiting embodiments or aspects, the sensitivity measure of the unsupervised machine learning model to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset.

[0025] According to some non-limiting embodiments or aspects, a computer program product is provided for using aggregate learning with machine learning models to prevent model performance degradation. The computer program product includes at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium including one or more instructions, which, when executed by at least one processor, cause the at least one processor to receive a training dataset of a plurality of data instances, wherein each data instance includes a time series of data points. The one or more instructions further cause the at least one processor to add a time delay amount to one or more of the plurality of data instances to provide an augmented training dataset. The one or more instructions further cause the at least one processor to select a first plurality of supervised machine learning models and select a second plurality of unsupervised machine learning models. The one or more instructions further cause the at least one processor to train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset. The one or more instructions further cause the at least one processor to generate an aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The one or more instructions further cause the at least one processor to generate a runtime output of the aggregated machine learning model based on the runtime input of the aggregated machine learning model.

[0026] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to add the time delay amount to one or more of the plurality of data instances cause the at least one processor to add the time delay amount to a plurality of labeled data instances to provide a plurality of augmented labeled data instances. The one or more instructions further cause the at least one processor to add the time delay amount to a plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances. In some non-limiting embodiments or aspects, the augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0027] In some non-limiting embodiments or aspects, the one or more instructions further cause the at least one processor to perform an action based on the runtime output of the aggregated machine learning model.

[0028] In some non-limiting embodiments or aspects, the one or more instructions that cause the at least one processor to generate the aggregated machine learning model cause the at least one processor to generate the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The one or more instructions further cause the at least one processor to train the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0029] In some non-limiting embodiments or aspects, the one or more instructions further cause the at least one processor to add a perturbation to one or more additional data instances among the plurality of data instances. The one or more instructions further cause the at least one processor to add the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0030] In some non-limiting embodiments or aspects, the one or more instructions causing the at least one processor to select the first plurality of supervised machine learning models cause the at least one processor to determine a sensitivity metric to time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset. The one or more instructions further cause the at least one processor to select a subset of the set of supervised machine learning models based on the sensitivity metric to time delay for each supervised machine learning model. In some non-limiting embodiments or aspects, the one or more instructions causing the at least one processor to select the second plurality of unsupervised machine learning models cause the at least one processor to determine a sensitivity metric to time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset. The one or more instructions further cause the at least one processor to select a subset of the set of unsupervised machine learning models based on the sensitivity metric to time delay for each unsupervised machine learning model.

[0031] Other embodiments are described in the following numbered clauses:

[0032] Clause 1: A system for aggregate learning using machine learning models to prevent model performance degradation, the system comprising: at least one processor programmed or configured to: receive a training dataset of a plurality of data instances, wherein each data instance includes a time series of data points; add a time delay amount to one or more of the plurality of data instances to provide an augmented training dataset; select a first plurality of supervised machine learning models; select a second plurality of unsupervised machine learning models; train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset; generate an aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models; and generate a runtime output of the aggregated machine learning model based on runtime inputs of the aggregated machine learning model.

[0033] Clause 2: The system according to Clause 1, wherein when the time delay amount is added to one or more of the plurality of data instances to provide the augmented training dataset, the at least one processor is further programmed or configured to: add the time delay amount to the plurality of labeled data instances to provide a plurality of augmented labeled data instances; and add the time delay amount to the plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances; and wherein the augmented training dataset comprises the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0034] Clause 3: The system according to Clause 1 or 2, wherein the at least one processor is further programmed or configured to: perform actions based on the runtime output of the aggregated machine learning model.

[0035] Clause 4: A system according to any one of Clauses 1 to 3, wherein when the aggregated machine learning model is generated, the at least one processor is programmed or configured to: generate the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and train the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0036] Clause 5: A system according to any one of Clauses 1 to 4, wherein the at least one processor is further programmed or configured to: add a perturbation to one or more additional data instances among the plurality of data instances; and add the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0037] Clause 6: A system according to any one of Clauses 1 to 5, wherein when the first plurality of supervised machine learning models are selected, the at least one processor is programmed or configured to: determine a sensitivity measure of time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset; and select a subset of the set of supervised machine learning models based on the sensitivity measure of time delay for each supervised machine learning model; and wherein when the second plurality of unsupervised machine learning models are selected, the at least one processor is programmed or configured to: determine a sensitivity measure of time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset; and select a subset of the set of unsupervised machine learning models based on the sensitivity measure of time delay for each unsupervised machine learning model.

[0038] Clause 7: A system according to any one of Clauses 1 to 6, wherein the sensitivity measure of the supervised machine learning model to time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset and relative to the augmented training dataset; and wherein the sensitivity measure of the unsupervised machine learning model to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset.

[0039] Clause 8: A method for aggregate learning using machine learning models to prevent model performance degradation, the method comprising: receiving a training dataset of a plurality of data instances using at least one processor, wherein each data instance includes a time series of data points; adding a time delay amount to one or more of the plurality of data instances using at least one processor to provide an augmented training dataset; selecting a first plurality of supervised machine learning models using at least one processor; selecting a second plurality of unsupervised machine learning models using at least one processor; training the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset using at least one processor; generating an aggregated machine learning model using at least one processor based on the outputs of the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models; and generating a runtime output of the aggregated machine learning model using at least one processor based on runtime inputs of the aggregated machine learning model.

[0040] Clause 9: The system according to Clause 8, wherein adding the time delay amount to one or more of the plurality of data instances to provide the augmented training dataset comprises: adding the time delay amount to a plurality of labeled data instances to provide a plurality of augmented labeled data instances; and adding the time delay amount to a plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances; and wherein the augmented training dataset comprises the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0041] Clause 10: The method described in Clause 8 or 9 further includes performing actions based on the runtime output of the aggregated machine learning model.

[0042] Clause 11: The method according to any one of Clauses 8 to 10, wherein generating the aggregated machine learning model comprises: generating the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and training the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0043] Clause 12: The method according to any one of Clauses 8 to 11 further comprises: adding a perturbation to one or more additional data instances among the plurality of data instances; and adding the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0044] Clause 13: The method according to any one of Clauses 8 to 12, wherein selecting the first plurality of supervised machine learning models comprises: determining a sensitivity measure of time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset; and selecting a subset of the set of supervised machine learning models based on the sensitivity measure of time delay for each supervised machine learning model; and wherein selecting the second plurality of unsupervised machine learning models comprises: determining a sensitivity measure of time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset; and selecting a subset of the set of unsupervised machine learning models based on the sensitivity measure of time delay for each unsupervised machine learning model.

[0045] Clause 14: The method according to any one of Clauses 8 to 13, wherein the sensitivity measure of the supervised machine learning model to time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset and relative to the augmented training dataset; and wherein the sensitivity measure of the unsupervised machine learning model to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset.

[0046] Clause 15: A computer program product for using aggregate learning with machine learning models to prevent model performance degradation, the computer program product comprising at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium comprising one or more instructions, the one or more instructions, when executed by at least one processor, causing the at least one processor to: receive a training dataset of a plurality of data instances, wherein each data instance comprises a time series of data points; add a time delay amount to one or more of the plurality of data instances to provide an augmented training dataset; select a first plurality of supervised machine learning models; select a second plurality of unsupervised machine learning models; train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset; generate an aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models; and generate a runtime output of the aggregated machine learning model based on runtime inputs of the aggregated machine learning model.

[0047] Clause 16: A computer program product according to Clause 15, wherein the one or more instructions that cause the at least one processor to add the time delay amount to one or more of the plurality of data instances cause the at least one processor to: add the time delay amount to the plurality of labeled data instances to provide a plurality of augmented labeled data instances; and add the time delay amount to the plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances; and wherein the augmented training dataset comprises the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0048] Clause 17: A computer program product pursuant to Clause 15 or 16, wherein one or more instructions further cause the at least one processor to perform an action based on the runtime output of the aggregated machine learning model.

[0049] Clause 18: A computer program product pursuant to any one of Clauses 15 to 17, wherein the one or more instructions that cause the at least one processor to generate the aggregated machine learning model cause the at least one processor to: generate the output of the aggregated machine learning model based on the output of the first plurality of supervised machine learning models and the output of the second plurality of unsupervised machine learning models; and train the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model.

[0050] Clause 19: A computer program product according to any one of Clauses 15 to 18, wherein the one or more instructions further cause the at least one processor to: add a perturbation to one or more additional data instances among the plurality of data instances; and add the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0051] Clause 20: A computer program product according to any one of Clauses 15 to 19, wherein the one or more instructions causing the at least one processor to select the first plurality of supervised machine learning models cause the at least one processor to: determine a sensitivity measure of time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset; and select a subset of the set of supervised machine learning models based on the sensitivity measure of time delay for each supervised machine learning model; and wherein the one or more instructions causing the at least one processor to select the second plurality of unsupervised machine learning models cause the at least one processor to: determine a sensitivity measure of time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset; and select a subset of the set of unsupervised machine learning models based on the sensitivity measure of time delay for each unsupervised machine learning model.

[0052] The operational methods and manufacturing economics of these and other features and characteristics of this disclosure, as well as the combinations of related structural elements and parts, will become more apparent after considering the following description and appended claims with reference to the accompanying drawings, all of which form part of this specification, wherein similar reference numerals in the drawings indicate corresponding parts. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be construed as limiting the scope of this disclosure. Unless the context clearly requires otherwise, the singular forms “a” and “described” as used in this specification and claims include plural indicators. Attached Figure Description

[0053] Figure 1 The figures are non-limiting embodiments or aspects of environments in which the systems, apparatuses, products, devices and / or methods described herein can be implemented according to the principles of this disclosure;

[0054] Figure 2 yes Figure 1 A diagram of a non-limiting embodiment or aspect of a component of one or more devices and / or one or more systems;

[0055] Figure 3 A flowchart illustrating a non-limiting embodiment or aspect of the process of generating machine learning models based on convergent learning methods using supervised and unsupervised techniques; and

[0056] Figures 4A-4E It is a diagram of a non-limiting embodiment or aspect of a process for generating machine learning models based on aggregation learning methods using supervised and unsupervised techniques. Detailed Implementation

[0057] For descriptive purposes, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives are intended to refer to the orientation of this disclosure as shown in the accompanying drawings. However, it should be understood that this disclosure may take various alternative variations and sequences of steps, except where explicitly specified otherwise. It should also be understood that the specific apparatus and processes shown in the drawings and described in the following specification are merely exemplary embodiments of this disclosure. Therefore, unless otherwise indicated, specific dimensions and other physical characteristics relating to embodiments of the embodiments disclosed herein should not be considered limiting.

[0058] The aspects, components, elements, structures, actions, steps, functions, instructions, etc., used herein should not be construed as critical or essential unless explicitly stated otherwise. Furthermore, as used herein, the article “a” is intended to include one or more items and is interchangeable with “one or more” and “at least one.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and is interchangeable with “one or more” or “at least one.” Where only one item is desired, the term “a” or similar language is used. Also, as used herein, the terms “having” and similar terms are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Where appropriate, the phrase “based on” may also mean “in response to.”

[0059] As used herein, the terms "communication" and "transmission" can refer to the receipt, acceptance, sending, transmission, provisioning, etc., of information (e.g., data, signals, messages, instructions, commands, etc.). Communication between one unit (e.g., a device, system, component of a device or system, combination thereof, etc.) and another unit means that the first unit is able to receive information directly or indirectly from and / or transmit (e.g., send) information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless. Furthermore, although the transmitted information may be modified, processed, relayed, and / or routed between the first and second units, the two units can also communicate with each other. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit (e.g., a third unit located between the first and second units) processes information received from the first unit and sends the processed information to the second unit. In some non-limiting embodiments, a message can refer to a network data packet (e.g., a data packet, etc.) that includes data.

[0060] As used herein, the terms “issuer,” “issuer institution,” “issuer bank,” or “payment device issuer” can refer to one or more entities that provide accounts for payment transactions, such as credit card payment transactions and / or debit card payment transactions, to individuals (e.g., users, customers, etc.). For example, an issuer institution may provide a customer with an account identifier, such as a primary account number (PAN), that uniquely identifies one or more accounts associated with said customer. In some non-limiting embodiments, an issuer may be associated with a bank identification number (BIN) that uniquely identifies an issuer institution. As used herein, “issuer system” can refer to one or more computer systems operated by or on behalf of an issuer, such as a server executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0061] As used herein, the term "transaction service provider" can refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment guarantees through an agreement between the transaction service provider and the issuing authority. For example, a transaction service provider may include a payment network, such as... Ameri can Or any other entity that processes transactions. As used herein, the term "transaction service provider system" can refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction service provider system executing one or more software applications. A transaction service provider system may include one or more processors and, in some non-limiting embodiments, may be operated by or on behalf of a transaction service provider.

[0062] As used herein, the term "merchant" can refer to one or more entities (e.g., operators of retail businesses) that provide goods and / or services and / or access to goods and / or services to users (e.g., customers, consumers) based on transactions such as payment transactions. As used herein, "merchant system" can refer to one or more computer systems operated by or on behalf of a merchant, such as servers executing one or more software applications. As used herein, the term "product" can refer to one or more goods and / or services provided by a merchant.

[0063] As used herein, the term "acquiring party" can refer to an entity authorized and approved by a transaction service provider to initiate transactions (e.g., payment transactions) involving payment devices associated with the transaction service provider. As used herein, the term "acquiring party system" can also refer to one or more computer systems, computer devices, etc., operated by or on behalf of the acquiring party. Transactions that an acquiring party can initiate may include payment transactions (e.g., purchases, Original Credit Transactions (OCT), Account Funds Transactions (AFT), etc.). In some non-limiting embodiments, the acquiring party may be authorized by the transaction service provider to contract with merchants or service providers to initiate transactions involving payment devices associated with the transaction service provider. The acquiring party may contract with payment service providers to enable them to provide sponsorships to merchants. The acquiring party may monitor the compliance of payment service providers in accordance with transaction service provider regulations. The acquiring party may conduct due diligence on payment service providers and ensure that appropriate due diligence occurs before contracting with sponsored merchants. The acquiring party may be responsible for all transaction service provider programs operated or sponsored by the acquiring party. The acquiring party may be responsible for the actions of its payment service provider, merchants sponsored by the payment service provider, etc. In some non-limiting embodiments, the acquiring party may be a financial institution, such as a bank.

[0064] As used herein, the term "payment gateway" can refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, payment service provider, payment servicer, payment servicer contracted with an acquirer, payment aggregator, etc.) that provides payment services (e.g., transaction service provider payment services, payment processing services, etc.) to one or more merchants. Payment services may be associated with the use of portable financial devices managed by a transaction service provider. As used herein, the term "payment gateway system" can refer to one or more computer systems, computer devices, servers, server clusters, etc., operated by or on behalf of a payment gateway.

[0065] As used herein, the terms "client" and "client device" can refer to one or more computing devices, such as processors, storage devices, and / or similar computer components that access services available from a server. In some non-limiting embodiments, a client device can include an electronic device configured to communicate with one or more networks and / or facilitate payment transactions, such as, but not limited to, one or more desktop computers, one or more portable computers (e.g., tablet computers), one or more mobile devices (e.g., cellular phones, smartphones, personal digital assistants, wearable devices such as watches, glasses, lenses, and / or clothing), and / or other similar devices. Furthermore, the term "client" can also refer to an entity that owns, uses, and / or operates a client device to facilitate transactions with another entity.

[0066] As used herein, the term "server" can refer to one or more computing devices, such as processors, storage devices, and / or similar computer components, that communicate with client devices and / or other computing devices on a network such as the Internet or a private network, and in some examples, facilitate communication between other servers and / or client devices.

[0067] As used herein, the term "system" may refer to one or more computing devices or combinations of computing devices, such as, but not limited to, processors, servers, client devices, software applications, and / or other similar components. Furthermore, as used herein, references to "server" or "processor" may refer to the server and / or processor previously stated to perform the preceding steps or functions, different servers and / or processors, and / or combinations of servers and / or processors. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different servers and / or processors stated to perform a second step or function.

[0068] Systems, methods, and computer program products are provided for generating machine learning models based on aggregation learning methods using supervised and unsupervised techniques. Embodiments of this disclosure may include a system for using aggregation learning with machine learning models to prevent model performance degradation. The system includes at least one processor programmed or configured to: receive a training dataset of multiple data instances, each data instance comprising a time series of data points; add a time delay amount to one or more of the multiple data instances to provide an augmented training dataset; select a first plurality of supervised machine learning models; select a second plurality of unsupervised machine learning models; train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models based on the augmented training dataset; generate an aggregation machine learning model based on the outputs of the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models; and generate a runtime output of the aggregation machine learning model based on runtime inputs to the aggregation machine learning model.

[0069] In some non-limiting embodiments or aspects, when the time delay amount is added to one or more of the plurality of data instances to provide the augmented training dataset, the at least one processor is further programmed and / or configured to add the time delay amount to a plurality of labeled data instances to provide a plurality of augmented labeled data instances. The at least one processor is further programmed and / or configured to add the time delay amount to a plurality of unlabeled data instances to provide a plurality of augmented unlabeled data instances. The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances.

[0070] In some non-limiting embodiments or aspects, the at least one processor is further programmed and / or configured to perform actions based on the runtime output of the aggregated machine learning model. In some non-limiting embodiments or aspects, when generating the aggregated machine learning model, the at least one processor is programmed and / or configured to generate the output of the aggregated machine learning model based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. The at least one processor is further programmed and / or configured to train the aggregated machine learning model using backpropagation based on the output of the aggregated machine learning model. In some non-limiting embodiments or aspects, the at least one processor is further programmed and / or configured to add perturbations to one or more additional data instances among the plurality of data instances. The at least one processor is further programmed and / or configured to add the one or more additional data instances among the plurality of data instances to the augmented training dataset.

[0071] In some non-limiting embodiments or aspects, when the first plurality of supervised machine learning models are selected, the at least one processor is programmed and / or configured to determine a sensitivity metric to time delay for each supervised machine learning model in a set of supervised machine learning models based on the augmented training dataset. The at least one processor is further programmed and / or configured to select a subset of the set of supervised machine learning models based on the sensitivity metric to time delay for each supervised machine learning model. In some non-limiting embodiments or aspects, when the second plurality of unsupervised machine learning models are selected, the at least one processor is programmed and / or configured to determine a sensitivity metric to time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on the augmented training dataset. The at least one processor is further programmed and / or configured to select a subset of the set of unsupervised machine learning models based on the sensitivity metric to time delay for each unsupervised machine learning model. In some non-limiting embodiments or aspects, the sensitivity metric to time delay for the supervised machine learning model includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset compared to the augmented training dataset. In some non-limiting embodiments or aspects, the sensitivity measure of the unsupervised machine learning model to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset compared to the augmented training dataset.

[0072] In this manner, embodiments or aspects of this disclosure combine supervised and unsupervised learning to generate machine learning models with improved accuracy and reduced model degradation over time while maintaining model freshness. Aggregate learning systems can use data, including data latency, to build robust and accurate deep learning models. In some non-limiting embodiments or aspects, model degradation can be reduced or eliminated when new patterns emerge in the model input data due to sudden changes in the market or technology (e.g., technology change, domain change, market change, etc.). In some non-limiting embodiments or aspects, the machine learning model may include a diverse set of base models and multiple clustering models to maintain model freshness and generality. In some non-limiting embodiments or aspects, data latency can be introduced to maintain model freshness, reduce degradation, improve the model's resilience to data changes and / or data latency, and improve the model's accuracy over time.

[0073] Now for reference Figure 1 , Figure 1 This is a diagram of an example environment 100 in which the apparatus, systems, methods, and / or products described herein can be implemented. Figure 1As shown, environment 100 includes an aggregate learning system 102, a user device 104, a data source 106, and a communication network 108. The aggregate learning system 102, user device 104, and data source 106 can be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing a communication connection).

[0074] The aggregation learning system 102 may include one or more computing devices configured to communicate with user device 104 and / or data source 106 via communication network 108. For example, the aggregation learning system 102 may include server clusters and / or other similar devices. In some non-limiting embodiments or aspects, the aggregation learning system 102 may be associated with (operated by) a transaction service provider as described herein. Alternatively or additionally, the aggregation learning system 102 may be a component of a transaction service provider system, an issuer system, and / or a merchant system. In some non-limiting embodiments or aspects, the aggregation learning system 102 may include one or more machine learning models. Unsupervised and / or supervised methods may be used to train the machine learning models. In some non-limiting embodiments or aspects, a dataset from data source 106 may be used to train the machine learning models. In some non-limiting embodiments or aspects, the output from one machine learning model may be used as input for training other machine learning models.

[0075] User device 104 may include one or more computing devices configured to communicate with aggregate learning system 102 and / or data source 106 via communication network 108. For example, user device 104 may include a desktop computer (e.g., a client device communicating with a server), a mobile device, etc. In some non-limiting embodiments or aspects, user device 104 may be associated with a user (e.g., an individual operating the device).

[0076] Data source 106 may include one or more datasets for training one or more machine learning models. In some non-limiting embodiments or aspects, data source 106 may include one or more static training datasets and / or one or more real-time training datasets. For example, data source 106 may include real-time training datasets that can be continuously updated with new data. In some non-limiting embodiments or aspects, data source 106 may include static training datasets that have been previously compiled and stored in data source 106. In this way, the static training dataset may not receive new data. Data source 106 may be updated via communication network 108. Data source 106 may be configured to communicate with aggregation learning system 102 and / or user device 104 via communication network 108. In some non-limiting embodiments or aspects, data source 106 may be updated with new data from one or more machine learning models. For example, the output from one or more machine learning models may be transmitted to data source 106 for storage. In some non-limiting embodiments or aspects, the output from one or more machine learning models stored in data source 106 may be used as input for one or more other machine learning models for future training.

[0077] The communication network 108 may include one or more wired and / or wireless networks. For example, the communication network 108 may include cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN)), Private Networks, Ad Hoc Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of these or other types of networks.

[0078] Provided as an example Figure 1 The number and arrangement of systems and / or devices are shown. Additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, or devices may exist in conjunction with... Figure 1 The systems and / or devices shown are arranged in different ways. Furthermore, they can be implemented within a single system and / or a single device. Figure 1 The two or more systems and / or devices shown, or Figure 1 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively, a group of systems or devices in environment 100 (e.g., one or more systems, one or more devices) may perform one or more functions described as being performed by another group of systems or devices in environment 100.

[0079] Now for reference Figure 2 , Figure 2 This is a diagram of example components of device 200. Device 200 may correspond to aggregation learning system 102 (e.g., one or more devices of aggregation learning system 102), user device 104, and / or data source 106. In some non-limiting embodiments or aspects, aggregation learning system 102, user device 104, and / or data source 106 may include at least one device 200. Figure 2 As shown, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212 and communication interface 214.

[0080] Bus 202 may include components that enable communication between components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, software, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessor, digital signal processor (DSP), and / or any processing component that can be programmed to perform functions (e.g., a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 206 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0081] Storage component 208 may store information and / or software associated with the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.), a compressed optical disk (CD), a digital versatile optical disk (DVD), a floppy disk, a cassette tape, a magnetic tape, and / or another type of computer-readable medium, and a corresponding drive. In some non-limiting embodiments or aspects, storage component 208 may correspond to data source 106.

[0082] Input component 210 may include components that allow device 200 to receive information, such as via user input (e.g., touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, camera, etc.). Alternatively, input component 210 may include sensors for sensing information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, etc.). Output component 212 may include components that provide output information from device 200 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.).

[0083] Communication interface 214 may include transceiver components (e.g., transceiver, separate receiver and transmitter, etc.) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may allow device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. interface, interface, Interfaces, cellular network interfaces, etc.

[0084] Apparatus 200 can perform one or more processes described herein. Apparatus 200 can perform these processes based on software instructions stored in a computer-readable medium, such as memory 206 and / or storage component 208, executed by processor 204. Computer-readable medium (e.g., non-transient computer-readable medium) is defined herein as a non-transient memory device. A non-transient memory device includes memory space located within a single physical storage device or memory space distributed across multiple physical storage devices.

[0085] Software instructions may be read into memory 206 and / or storage component 208 via communication interface 214 from another computer-readable medium or from another device. When executed, the software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Alternatively or additionally, hard-wired circuitry may be used in place of or in conjunction with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0086] The memory 206 and / or storage component 208 may include a data storage device or one or more data structures (e.g., a database). The device 200 is capable of receiving information from the data storage device or one or more data structures in the memory 206 and / or storage component 208, storing information in the data storage device or one or more data structures, conveying information to the data storage device or one or more data structures, or searching for information stored therein. For example, the information may include input data, output data, transaction data, account data, or any combination thereof.

[0087] supply Figure 2 The number and arrangement of components shown are for illustrative purposes only. In some non-limiting embodiments or aspects, with Figure 2Compared to those shown, device 200 may include additional components, fewer components, different components, or components arranged in a different manner. Alternatively, a set of components of device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 200.

[0088] Now for reference Figure 3 , Figure 3 This is a flowchart of a non-limiting embodiment or aspect of a process 300 for generating a machine learning model based on a convergent learning method using supervised and unsupervised techniques. In some non-limiting embodiments or aspects, one or more functions described regarding process 300 may be performed by the convergent learning system 102 (e.g., fully, partially, etc.). In some non-limiting embodiments or aspects, one or more steps of process 300 may be performed by another device or group of devices (e.g., user device 104) (e.g., fully, partially, etc.) that is separate from or includes the convergent learning system 102.

[0089] like Figure 3 As shown, in step 302, process 300 may include receiving a training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 may receive a training dataset of multiple data instances, wherein each data instance includes a time series of data points. For example, the aggregation learning system 102 may receive real-time and / or static training datasets from data source 106 for training one or more machine learning models. In some non-limiting embodiments or aspects, the static training dataset may include a predefined training dataset stored in a storage component and / or stored in data source 106. In some non-limiting embodiments or aspects, the aggregation learning system 102 may receive the training dataset in real-time as a time-ordered dataset while data is being collected. In some non-limiting embodiments or aspects, the aggregation learning system 102 may receive a training dataset corresponding to the outputs from one or more machine learning models.

[0090] In some non-limiting embodiments or aspects, each data instance in the training dataset may represent an event, such as an electronic payment transaction (e.g., electronic credit card payment transaction, electronic debit card payment transaction, etc.) between a user associated with user device 104 and a merchant associated with a merchant system. In some non-limiting embodiments or aspects, each data instance may be received in real-time relative to the event. In some non-limiting embodiments or aspects, each data instance may be stored and compiled into a training dataset for future training. In some non-limiting embodiments or aspects, each data instance may include transaction data associated with the electronic payment transaction. In some non-limiting embodiments or aspects, the transaction data may include transaction parameters associated with the electronic payment transaction. Transaction parameters may include: e-wallet card data associated with the e-card (e.g., e-credit card, e-debit card, e-loyalty card, etc.), decision data associated with the decision (e.g., decision to approve or reject a transaction authorization request), authorization data associated with the authorization response (e.g., approved spending limit, approved transaction value, etc.), PAN, authorization code (e.g., PIN, etc.), data associated with the transaction amount (e.g., approved limit, transaction value, etc.), data associated with the transaction date and time, data associated with the currency exchange rate, data associated with the merchant type (e.g., goods, groceries, fuel, etc.), data associated with the acquiring institution's country, data associated with the identifier of the country associated with the PAN, data associated with the response code, data associated with the merchant identifier (e.g., merchant name, merchant location, etc.), and data associated with the currency type corresponding to the funds stored in association with the PAN, etc.

[0091] In some non-limiting embodiments or aspects, the predicted classification value may include the amount of time associated with the event. For example, the predicted classification value may represent the amount of time taken to complete the event. In one example, the predicted classification value may represent the amount of time (e.g., days) taken to clear (e.g., the process involving the actual movement of electronic funds from one account to another, such as in the form of an electronic payment request) an electronic payment transaction between a user associated with user device 104 and a merchant associated with the merchant system.

[0092] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate a training dataset based on the outputs of one or more trained machine learning models. For example, the aggregation learning system 102 can provide each data instance of the dataset as input to the trained machine learning model, and the aggregation learning system 102 can generate the output of the trained machine learning model as input for each data instance of the dataset.

[0093] In some non-limiting embodiments or aspects, the aggregation learning system 102 can determine the classification (e.g., predicted classification, initial predicted classification, etc.) of each data instance in the dataset. For example, the aggregation learning system 102 can determine the classification of the initial input by providing it to one or more trained machine learning models (e.g., trained machine learning models including random forests, multilayer perceptrons, and / or neural networks such as deep neural networks), and determine the classification as the output from the machine learning models. In some non-limiting embodiments or aspects, the one or more trained machine learning models can be machine learning classifiers including deep learning networks. In some non-limiting embodiments or aspects, the classification can be associated with a category that includes a group of members, and the classification can refer to a feature shared among members of the group within the category. In some non-limiting embodiments or aspects, the aggregation learning system 102 can store the training dataset in a database.

[0094] In some non-limiting embodiments or aspects, each of the plurality of values ​​associated with the prediction matrix may include an error value representing the difference between a predicted classification value (e.g., a time quantity, such as days) of one or more data instances and a ground truth value of one or more data instances. The values ​​of the prediction matrix may include error values ​​between a first ground truth value of one or more data instances and a first predicted classification value of one or more data instances, error values ​​between a second ground truth value of one or more data instances and a first predicted classification value of one or more data instances, error values ​​between a first ground truth value of one or more data instances and a second predicted classification value of one or more data instances, error values ​​between a third ground truth value of one or more data instances and a first predicted classification value of one or more data instances, error values ​​between a first ground truth value of one or more data instances and a third predicted classification value of one or more data instances, error values ​​between a third ground truth value of one or more data instances and a second predicted classification value of one or more data instances, error values ​​between a second ground truth value of one or more data instances and a third predicted classification value of one or more data instances, and so on. In some non-limiting embodiments or aspects, the plurality of values ​​associated with the prediction matrix may include upper error values ​​(e.g., upper diagonal error values) and lower error values ​​(e.g., lower diagonal error values) for a plurality of events. In some non-limiting embodiments or aspects, the upper error value includes an error value associated with the predicted classification value of the event being greater than the true value of the event. In some non-limiting embodiments or aspects, the lower error value may include an error value associated with the predicted classification value of the event being less than the true value of the event. In some non-limiting embodiments or aspects, the plurality of values ​​associated with the prediction matrix may include correct prediction values, which may include values ​​associated with the predicted classification value of the event being equal to the true value of the event.

[0095] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate one or more trained machine learning models. For example, the aggregation learning system 102 can generate trained machine learning models based on a training dataset to provide predictive classifications of data instances, such as data instances associated with events.

[0096] In some non-limiting embodiments or aspects, one or more trained machine learning models may include machine learning models designed to receive data instances associated with an event as input and provide a predicted classification of the data instances as output. In some non-limiting embodiments or aspects, the aggregation learning system 102 may store trained machine learning models (e.g., for later use).

[0097] In some non-limiting embodiments or aspects, as described herein, the aggregation learning system 102 may process data instances associated with an event (e.g., historical data instances associated with the event) to obtain training data (e.g., a training dataset) for a machine learning model. For example, the aggregation learning system 102 may process data to transform it into a format that can be analyzed (e.g., by the aggregation learning system 102) to generate one or more trained machine learning models. The transformed data (e.g., the data resulting from the transformation) may be referred to as training data. In some non-limiting embodiments or aspects, the aggregation learning system 102 may process event-associated data instances based on received data instances to obtain training data. Additionally or alternatively, the aggregation learning system 102 may process data to obtain training data based on instructions received from a user of the aggregation learning system 102 (e.g., a user associated with user device 104) indicating that the aggregation learning system 102 will process data, such as when the aggregation learning system 102 receives an instruction to generate a machine learning model for predicting the classification of events.

[0098] In some non-limiting embodiments or aspects, the aggregation learning system 102 can process data instances associated with events by determining predictor variables based on data. Predictor variables may include metrics associated with the event, which can be derived from data instances associated with the event. Predictor variables can be analyzed to generate trained machine learning models. For example, predictor variables may include variables associated with event time, variables associated with event parameters, variables associated with the frequency of events, etc.

[0099] In some non-limiting embodiments or aspects, the aggregation learning system 102 can analyze training data to generate one or more trained machine learning models. For example, the aggregation learning system 102 can use machine learning techniques to analyze training data to generate trained machine learning models. In some non-limiting embodiments or aspects, generating a trained machine learning model (e.g., based on training data) can be referred to as training a machine learning model. Machine learning techniques can include, for example, supervised and / or unsupervised techniques, such as decision trees, random forests, logistic regression, linear regression, gradient boosting, support vector machines, additional trees (e.g., extensions of random forests), Bayesian statistics, learning automata, implicit Markov modeling, linear classifiers, quadratic classifiers, association rule learning, etc. In some non-limiting embodiments or aspects, the machine learning model can include a model specific to specific characteristics, such as a model specific to a specific entity involved in an event, a specific time interval in which the event occurs, etc. Alternatively or additionally, the machine learning model can be specifically targeted at a specific entity involved in the event (e.g., a business entity such as a merchant, a consumer entity such as an account holder of an account issued by an issuer, the issuer, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate one or more trained machine learning models for one or more entities, a specific group of entities, and / or one or more users of one or more entities.

[0100] Alternatively or concurrently, when analyzing training data, the aggregation learning system 102 may identify one or more variables (e.g., one or more independent variables) as predictor variables (e.g., features) that can be used to make predictions when analyzing the training data. In some non-limiting embodiments or aspects, the values ​​of the predictor variables may be inputs to a machine learning model. For example, the aggregation learning system 102 may identify a subset of variables (e.g., an appropriate subset) as predictor variables that can be used to accurately predict event classifications. In some non-limiting embodiments or aspects, as discussed above, the predictor variables may include one or more predictor variables that have a significant impact (e.g., an impact that meets a threshold) on the predicted classification of events, as determined by the aggregation learning system 102.

[0101] In some non-limiting embodiments or aspects, the aggregation learning system 102 can validate the machine learning model. For example, after the aggregation learning system 102 generates a machine learning model, it can validate the machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 can validate the machine learning model based on a portion of the training data to be used for validation. For example, the aggregation learning system 102 can divide the training data into a first portion and a second portion, wherein the first portion can be used to generate the machine learning model, as described above.

[0102] In some non-limiting embodiments or aspects, the aggregation learning system 102 can validate the machine learning model by providing validation data associated with a user (e.g., data associated with one or more events involving the user) as input to the machine learning model, and determining whether the machine learning model correctly or incorrectly predicts the classification of events based on the output of the machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 can validate the machine learning model based on a validation threshold. For example, the aggregation learning system 102 can be configured to validate the machine learning model when it correctly predicts the classification of multiple events (as identified by validation data) (e.g., when the machine learning model correctly predicts 50% of the classifications of multiple events, 70% of the classifications of multiple events, a threshold amount for the classification of multiple events, etc.).

[0103] In some non-limiting embodiments or aspects, if the aggregation learning system 102 does not validate the machine learning model (e.g., when the percentage of correctly predicted classifications of multiple events does not meet a validation threshold), the aggregation learning system 102 may generate one or more additional machine learning models.

[0104] In some non-limiting embodiments or aspects, once the machine learning model has been validated, the aggregation learning system 102 can further train the machine learning model and / or generate new machine learning models based on received new training data. The new training data may include additional data associated with one or more events. In some non-limiting embodiments or aspects, the new training data may include data associated with multiple additional events. The aggregation learning system 102 can use the machine learning model to predict the classification of the multiple additional events and compare the output of the machine learning model with the new training data. In such examples, the aggregation learning system 102 may update one or more trained machine learning models based on the new training data.

[0105] In some non-limiting embodiments or aspects, the aggregation learning system 102 may store trained machine learning models. For example, the aggregation learning system 102 may store trained machine learning models in a data structure (e.g., a database, linked list, tree, etc.). The data structure may be located within the aggregation learning system 102 or outside the aggregation learning system 102 (e.g., remote from the aggregation learning system).

[0106] like Figure 3As shown, in step 304, process 300 may include adding a perturbation. In some non-limiting embodiments or aspects, the aggregation learning system 102 may add a perturbation to one or more data instances among a plurality of data instances to provide an augmented training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 may add a time delay amount to one or more data instances among a plurality of data instances to provide an augmented training dataset. For example, the aggregation learning system 102 may add a time delay to one or more data instances among a plurality of data instances by inserting new and / or pre-existing data instances into the plurality of data instances at specified time intervals. Alternatively or additionally, the time interval for inserting new and / or pre-existing data instances into the plurality of data instances may be random. In some non-limiting embodiments or aspects, the aggregation learning system 102 may add a time delay to one or more data instances among a plurality of data instances by removing data instances from the plurality of data instances at specified time intervals. Alternatively or additionally, the time interval for removing data instances from the plurality of data instances may be random. In some non-limiting embodiments or aspects, the time delay may be generated continuously within the plurality of data instances. Alternatively, a time delay can be generated at a specific interval from a time equal to 0 to a selected upper limit.

[0107] In some non-limiting embodiments or aspects, the aggregation learning system 102 can add a time delay to one or more data instances among a plurality of data instances by comparing the current real-time with a predetermined time of an event associated with the data instance. In some non-limiting embodiments or aspects, the current real-time may include time measured at the moment an action is performed (e.g., when a time delay is added, when a model is trained on a training dataset, when the model reads a data instance, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 may add a perturbation to one or more additional data instances among the plurality of data instances. In some non-limiting embodiments or aspects, the aggregation learning system 102 may add one or more additional data instances among the plurality of data instances to an augmented training dataset. In some non-limiting embodiments or aspects, the perturbation may include small changes added to one or more data instances among the plurality of data instances (e.g., adding noise, data distortion, new data sequences, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 may use adversarial training techniques (e.g., poisoning, fast gradient signification, visual similarity, semantic similarity, etc.) to add perturbations to one or more data instances.

[0108] like Figure 3As shown, in step 306, process 300 may include selecting a machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a first plurality of supervised machine learning models. For example, the aggregation learning system 102 may select a first plurality of trained and / or untrained supervised machine learning models based on the classification outputs of a plurality of supervised models. In some non-limiting embodiments or aspects, the output of the trained machine learning model may include predicted classification values ​​of events provided as input, such as real-time events. In some non-limiting embodiments or aspects, the output of the trained supervised machine learning model may include predicted classification values ​​of events, wherein the events are not associated with data instances of the dataset used to train the trained supervised machine learning model.

[0109] In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a first plurality of supervised machine learning models by determining a sensitivity metric for each supervised machine learning model in a set of supervised machine learning models to perturbations such as time delay based on an augmented training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a subset of supervised machine learning models based on a sensitivity metric for time delay of each supervised machine learning model. In some non-limiting embodiments or aspects, the sensitivity metric for time delay of the machine learning model (e.g., supervised or unsupervised machine learning model) may include a measure of the change in accuracy of the machine learning model relative to the training dataset compared to the augmented training dataset.

[0110] In some non-limiting embodiments or aspects, the aggregation learning system 102 may use a first plurality of supervised machine learning models to perform aggregation learning. In some non-limiting embodiments or aspects, the aggregation learning using the first plurality of supervised machine learning models with the aggregation learning system 102 may be based on a Bayesian optimal classifier, boosting, Bayesian model averaging, model bucketing, etc. Alternatively, the aggregation learning using the first plurality of supervised machine learning models may be based on stacking multiple machine learning models, wherein one or more machine learning models use the outputs of other machine learning models as input for training.

[0111] In some non-limiting embodiments or aspects, aggregation learning may include a machine learning process that trains one or more machine learning models (e.g., trained machine learning models including random forests, multilayer perceptrons, and / or neural networks such as deep neural networks) using the outputs from two or more machine learning models, said two or more machine learning models having been trained individually using the same training dataset as the inputs to said one or more machine learning models (e.g., the one or more machine learning models that have been trained). In some non-limiting embodiments or aspects, algorithms may be used to train the machine learning models during the aggregation learning process. For example, algorithms used during the aggregation learning process may include weighted averaging, superposition generalization (e.g., superposition), bootstrapping aggregation (e.g., bagging), boosting algorithms, etc.

[0112] In some non-limiting embodiments or aspects, the aggregation learning system 102 may receive data associated with a training dataset, and the aggregation learning system 102 may provide the data associated with the training dataset as input to one or more trained machine learning models. The aggregation learning system 102 may generate output from one or more trained machine learning models based on the input. In some non-limiting embodiments or aspects, the output from the trained supervised machine learning model may include multiple predicted classification values. The multiple predicted classification values ​​may include predicted classification values ​​for each category (e.g., each class label) that the trained machine learning model is configured to provide predictions for.

[0113] In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a second plurality of unsupervised machine learning models. For example, the aggregation learning system 102 may select a second plurality of trained and / or untrained unsupervised machine learning models based on the outputs of data clusters of multiple unsupervised models. In some non-limiting embodiments or aspects, the output of one or more trained unsupervised machine learning models may include predicted clustering values ​​of events and / or data instances provided as input, such as real-time events and / or data instances. In some non-limiting embodiments or aspects, the output of one or more trained unsupervised machine learning models may include predicted classification values ​​of events, wherein the events are not events associated with data instances of the dataset used to train the one or more unsupervised machine learning models.

[0114] In some non-limiting embodiments or aspects, the aggregation learning system 102 can select a second or more unsupervised machine learning models by determining a sensitivity measure of time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on an augmented training dataset. The aggregation learning system 102 can also select a subset of the set of unsupervised machine learning models based on the sensitivity measure of time delay for each unsupervised machine learning model.

[0115] In some non-limiting embodiments or aspects, the aggregation learning system 102 may use a second plurality of unsupervised machine learning models to perform aggregation learning. The aggregation learning of the aggregation learning system 102 using the second plurality of unsupervised machine learning models may be based on clustering models (e.g., combinatorial clustering models, co-associations, meta-clustering, exclusive clustering, etc.).

[0116] In some non-limiting embodiments or aspects, the aggregation learning system 102 may receive multiple data instances associated with a training dataset, and the aggregation learning system 102 may provide the multiple data instances associated with the training dataset as input to one or more trained machine learning models. The aggregation learning system 102 may generate output from one or more trained machine learning models based on the input. In some non-limiting embodiments or aspects, the output from one or more trained unsupervised machine learning models may include multiple cluster values. The multiple cluster values ​​may include the centroid values ​​(e.g., k-centers of k-means clustering) of each cluster for which the trained machine learning model is configured to provide clustered data predictions.

[0117] In some non-limiting embodiments or aspects, the aggregation learning system 102 can train a first plurality of supervised machine learning models and a second plurality of unsupervised machine learning models based on an augmented training dataset. For example, the aggregation learning system 102 can train the first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models by providing the augmented training dataset as input to each of the first plurality of supervised machine learning models and each of the second plurality of unsupervised machine learning models. In some non-limiting embodiments or aspects, the aggregation learning system 102 can provide the first plurality of machine learning models and the second plurality of machine learning models as input using an unmodified training dataset.

[0118] like Figure 3As shown, in step 308, process 300 may include generating an aggregated machine learning model. In some non-limiting embodiments or aspects, the aggregated learning system 102 may generate the aggregated machine learning model based on the outputs of a first plurality of supervised machine learning models and the outputs of a second plurality of unsupervised machine learning models. For example, the aggregated learning system 102 may generate the aggregated machine learning model by providing classification scores from each supervised machine learning model and clustering values ​​from each unsupervised machine learning model to another machine learning model for training the collected output values. In some non-limiting embodiments or aspects, the aggregated learning system 102 may generate the aggregated machine learning model through aggregated learning based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models. In some non-limiting embodiments or aspects, the generated aggregated model may be trained based on its own output using a feedback loop.

[0119] like Figure 3 As shown, in step 310, process 300 may include generating runtime output. In some non-limiting embodiments or aspects, the aggregation learning system 102 may generate runtime output of the aggregation machine learning model based on runtime input of the aggregation machine learning model. In some non-limiting embodiments or aspects, runtime output may include output generated by the aggregation learning system 102 when the aggregation learning system 102 is executed in a production environment. In some non-limiting embodiments or aspects, runtime input may include input received by the aggregation learning system 102 when the aggregation learning system 102 is executed in a production environment.

[0120] In some non-limiting embodiments or aspects, runtime may include the time period when the aggregate learning system 102 is executed in a production environment. For example, the aggregate learning system 102 may be in runtime when it is placed in operation intended for use by a user, and when it is being executed on production hardware for daily operation. In other words, runtime may include the time period when the aggregate learning system 102 is provided to clients as a real-time service. In some non-limiting embodiments or aspects, runtime may refer to the state of the aggregate learning system. For example, the aggregate learning system 102 may be in runtime (e.g., the aggregate learning system 102 is real-time) rather than in a development or testing state.

[0121] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate a final predictive output with the aggregation machine learning model in real time during runtime on transaction data inputs provided to the aggregation machine learning model. In some non-limiting embodiments or aspects, the accuracy of the aggregation machine learning model can be greater than the accuracy of individual supervised and unsupervised machine learning models. In some non-limiting embodiments or aspects, the accuracy of the aggregation machine learning model can be greater than the accuracy of individual supervised and unsupervised machine learning models over time.

[0122] In some non-limiting embodiments or aspects, real-time can include the moment when an event occurs (e.g., real-time regarding a transaction, real-time regarding data reading or writing, etc.), where a response can occur within a specified time period, typically a relatively short period. For example, real-time can refer to the moment when a disturbance is added to a data instance simultaneously with or shortly after the collection and storage of the corresponding data instance (e.g., in the millisecond range). As another example, a real-time output can be generated relative to real-time input received simultaneously with or within the millisecond range (e.g., approving a transaction simultaneously with or shortly after its initiation).

[0123] Now for reference Figures 4A-4E , Figures 4A-4E This is a diagram of an implementation scheme 400 of a process (e.g., process 300) that uses supervised and unsupervised techniques to generate machine learning models based on a convergent learning method. Figures 4A-4E As shown, implementation scheme 400 may include the steps of the aggregation learning system 102 performing the process.

[0124] like Figure 4A As indicated by reference numeral 510, the aggregation learning system 102 can receive a training dataset of multiple data instances. In some non-limiting embodiments or aspects, the aggregation learning system 102 can receive a pre-existing training dataset stored in a storage component. In some non-limiting embodiments or aspects, the aggregation learning system 102 can receive a training dataset generated in real time. For example, when image data is being collected, the aggregation learning system 102 can receive real-time image data (e.g., video data, medical imaging data, etc.). As another example, when a customer-initiated transaction is being processed, the aggregation learning system 102 can receive real-time transaction data.

[0125] In some non-limiting embodiments or aspects, the training dataset may include the output from one or more machine learning models. In some non-limiting embodiments or aspects, the training dataset may include a complete dataset or an incomplete dataset. In other non-limiting embodiments or aspects, the training dataset may include data from multiple different datasets.

[0126] like Figure 4BAs indicated by reference numeral 515, the aggregation learning system 102 can add a time delay amount to one or more data instances among a plurality of data instances. In some non-limiting embodiments or aspects, the aggregation learning system 102 can add a time delay amount to one or more data instances among a plurality of data instances by inserting new and / or pre-existing data instances into the plurality of data instances at specified time intervals. In some non-limiting embodiments or aspects, the time interval for inserting new and / or pre-existing data instances into the plurality of data instances can be random. In some non-limiting embodiments or aspects, the aggregation learning system 102 can add a time delay amount to one or more data instances among a plurality of data instances by removing data instances from the plurality of data instances at specified time intervals. In some non-limiting embodiments or aspects, the time interval for removing data instances from the plurality of data instances can be random. In some non-limiting embodiments or aspects, the time delay amount can be generated continuously within the plurality of data instances. Alternatively or additionally, the time delay can be generated at specific time intervals ranging from a time equal to 0 to a selected upper limit. The aggregation learning system 102 can add a time delay amount to one or more labeled data instances and / or one or more unlabeled data instances. In some non-limiting embodiments or aspects, the time delay amount may be input into one or more supervised models and / or one or more unsupervised models being trained.

[0127] In some non-limiting embodiments or aspects, the aggregation learning system 102 may add other perturbations to one or more data instances among a plurality of data instances. For example, the aggregation learning system 102 may add an amount of spatial deformation to image segments in segmented image data. As another example, the aggregation learning system 102 may perturb the model weights of a trained machine learning model. The aggregation learning system 102 may add other perturbations to the training dataset or machine learning model trained through aggregation learning by adding adversarial examples to one or more data instances among a plurality of data instances (e.g., modifying data, mislabeling data, adding noise, etc.).

[0128] like Figure 4CAs indicated by reference numeral 520, the aggregation learning system 102 can select a first plurality of supervised machine learning models. In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the first plurality of supervised machine learning models by determining a sensitivity measure to time delay for each supervised machine learning model in a set of supervised machine learning models based on an augmented training dataset. The aggregation learning system 102 can select a subset of the set of supervised machine learning models based on the sensitivity measure to time delay for each supervised machine learning model. In some non-limiting embodiments or aspects, the sensitivity measure to time delay for a supervised machine learning model can include a measure of the change in accuracy of the supervised machine learning model relative to the training dataset compared to the augmented training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the first plurality of supervised machine learning models based on supervised machine learning models that have different sensitivity measures to different data (e.g., based on PAN history, device or Internet Protocol (IP) address, account information, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the first plurality of supervised machine learning models based on the type of machine learning model algorithm (e.g., support vector machine (SVM), decision tree, neural network (NN), etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a first plurality of supervised machine learning models based on the size of the training dataset to be used to train multiple supervised machine learning models. In other non-limiting embodiments or aspects, the aggregation learning system 102 may select a first plurality of supervised machine learning models based on the processing speed of the models. In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a first plurality of supervised machine learning models based on the type of data instances in the training dataset, the domain of the training dataset, and / or the use cases associated with the training dataset (e.g., image classification, audio classification, text classification, etc.).

[0129] like Figure 4CAs indicated by reference numeral 525, the aggregation learning system 102 can select a second plurality of unsupervised machine learning models. In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the second plurality of unsupervised machine learning models by determining a sensitivity measure to time delay for each unsupervised machine learning model in a set of unsupervised machine learning models based on an augmented training dataset. The aggregation learning system 102 can select a subset of the set of unsupervised machine learning models based on the sensitivity measure to time delay for each unsupervised machine learning model. In some non-limiting embodiments or aspects, the sensitivity measure to time delay for the unsupervised machine learning model can include a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset compared to the augmented training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the second plurality of unsupervised machine learning models based on unsupervised machine learning models that have different sensitivity measures to different data. In some non-limiting embodiments or aspects, the aggregation learning system 102 can select the second plurality of unsupervised machine learning models based on the type of machine learning model algorithm (e.g., principal component analysis (PCA), autoencoder, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a second or more unsupervised machine learning models based on the size of the training dataset used to train the multiple unsupervised machine learning models. In other non-limiting embodiments or aspects, the aggregation learning system 102 may select a second or more unsupervised machine learning models based on the processing speed of the models. In some non-limiting embodiments or aspects, the aggregation learning system 102 may select a second or more unsupervised machine learning models based on the type of data instances in the training dataset, the domain of the training dataset, and / or the use cases associated with the training dataset (e.g., object identification, customer roles, product recommendations, etc.).

[0130] like Figure 4C As shown by reference numeral 530, the aggregation learning system 102 can train a first plurality of supervised machine learning models and a second plurality of unsupervised machine learning models based on an augmented training dataset. In some non-limiting embodiments or aspects, each of the first plurality of supervised machine learning models and each of the second plurality of unsupervised machine learning models can be trained using a single augmented training dataset. In some non-limiting embodiments or aspects, each of the plurality of machine learning models can be trained using different augmented training datasets. In some non-limiting embodiments or aspects, each of the plurality of machine learning models can be trained using one or more augmented training datasets.

[0131] In some non-limiting embodiments or aspects, the aggregation learning system 102 can train a first plurality of supervised machine learning models based on supervised learning algorithms, such as neural networks, Naive Bayes, linear regression, logistic regression, support vector machines, decision trees, random forests, etc. In some non-limiting embodiments or aspects, the aggregation learning system 102 can train the first plurality of supervised machine learning models using different types of data instances in the training dataset (e.g., based on PAN history, device or IP address, account information, etc.).

[0132] In some non-limiting embodiments or aspects, the aggregation learning system 102 can train a second or more unsupervised machine learning models based on master component analysis, autoencoders, single-value decomposition, k-means clustering, etc.

[0133] In some non-limiting embodiments or aspects, the aggregation learning system 102 can train multiple machine learning models using different training datasets with different amounts of time delay as input to each supervised and unsupervised machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 can train multiple machine learning models using training datasets with different sampling techniques to minimize data bias.

[0134] like Figure 4D As shown by reference numeral 535, the aggregation learning system 102 can generate an aggregation machine learning model based on the outputs of a first plurality of supervised machine learning models and the outputs of a second plurality of unsupervised machine learning models. In some non-limiting embodiments or aspects, the outputs of the first plurality of supervised machine learning models may include classification values. In some non-limiting embodiments or aspects, the classification values ​​may correspond to the performance of the supervised machine learning models. In some non-limiting embodiments or aspects, classification values ​​may be assigned to each of a plurality of data instances based on the performance of the supervised machine learning models on the output data. In some non-limiting embodiments or aspects, each supervised machine learning model may output one or more classification values ​​based on a plurality of data instances in the training dataset. In some non-limiting embodiments or aspects, the classification values ​​may be input into an aggregation machine learning model used for training to generate the aggregation machine learning model.

[0135] In some non-limiting embodiments or aspects, the output of the second plurality of unsupervised machine learning models may include data clusters (e.g., clusters of similar data objects). In some non-limiting embodiments or aspects, the data clusters may correspond to the performance of the unsupervised machine learning models. In some non-limiting embodiments or aspects, clusters may be assigned to each of the plurality of data instances based on the performance of the unsupervised machine learning models outputting the data. In some non-limiting embodiments or aspects, each unsupervised machine learning model may output one or more clusters. In some non-limiting embodiments or aspects, the data clusters may be input into an aggregated machine learning model for training to generate an aggregated machine learning model.

[0136] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate aggregation machine learning models based on Bayesian optimal classifiers, boosting algorithms, Bayesian model averaging, model bucketing, stacking, etc. In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate aggregation machine learning models based on combinations of clustering models (e.g., co-association, meta-clustering, etc.). In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate aggregation machine learning models based on the outputs of all supervised and unsupervised machine learning models that serve as input to the aggregation machine learning model. In other non-limiting embodiments or aspects, the aggregation learning system 102 can generate aggregation machine learning models based on the outputs of supervised and unsupervised machine learning models that serve as part of the input to the aggregation machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate aggregation machine learning models based on the outputs of other aggregation machine learning models.

[0137] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate the output of the aggregation machine learning model based on the outputs of a first plurality of supervised machine learning models and the outputs of a second plurality of unsupervised machine learning models.

[0138] like Figure 4DAs indicated by reference numeral 540, the aggregation learning system 102 can generate the runtime output of an aggregation machine learning model based on the runtime input of the aggregation machine learning model. In some non-limiting embodiments or aspects, the runtime output can be based on an added amount of time delay determined by comparing the current real-time with a predetermined time of an event associated with a data instance. In some non-limiting embodiments or aspects, the current real-time may include time measured at the moment an action is performed (e.g., when a time delay is added, when the model is trained on a training dataset, when the model receives a data instance, etc.). For example, when the aggregation learning system 102 generates the runtime output of the aggregation machine learning model based on the runtime input of the aggregation machine learning model, the aggregation learning system 102 can determine the amount of time delay by comparing the moment when the aggregation machine learning model reads an input data instance with the time of an event associated with the input data instance. As another example, the aggregation learning system 102 can determine the amount of time delay by comparing the time when the aggregation machine learning model receives a transaction data instance as input with the time when the transaction event occurs. The time when a transaction event occurs can be determined based on timestamps or transactions, etc. In some non-limiting embodiments or aspects, the generation of the runtime output of the aggregate machine learning model from the runtime input of the aggregate machine learning model can occur in real time relative to receiving the runtime input.

[0139] In some non-limiting embodiments or aspects, the aggregation learning system 102 can generate runtime outputs of an aggregation machine learning model, wherein the output of the aggregation machine learning model can include final predictions based on runtime inputs to the aggregation machine learning model. For example, the aggregation learning system 102 can generate predictions including transaction approvals (e.g., based on real-time transaction data received during runtime and input to the aggregation machine learning model). The runtime output of the prediction (Standby Processing (STIP)). In other non-limiting embodiments or aspects, the aggregation learning system 102 may generate a runtime output including fraudulent transaction predictions based on real-time transaction data input received during runtime. In some non-limiting embodiments or aspects, the aggregation learning system 102 may generate runtime outputs for the aggregation machine learning model in a feedback loop to provide additional input to the aggregation machine learning model. The aggregation learning system 102 may provide the feedback loop of the aggregation machine learning model as a continuous feedback loop. Alternatively, the aggregation learning system 102 may selectively provide specific instances of the output as feedback to the aggregation machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 may train the aggregation machine learning model using backpropagation based on the output of the aggregation machine learning model.

[0140] In some non-limiting embodiments or aspects, the aggregation learning system 102 can perform actions based on the runtime output of the aggregation machine learning model. For example, if the output prediction includes STIP transaction approval, the aggregation learning system 102 can approve or reject the transaction based on the runtime output of the aggregation machine learning model. In some non-limiting embodiments or aspects, the aggregation learning system 102 can perform actions in real time relative to the runtime output that generates the aggregation machine learning model.

[0141] like Figure 4E As indicated by reference numeral 545, the aggregation learning system 102 may add a time delay amount to one or more data instances among a plurality of data instances to provide an augmented training dataset. In some non-limiting embodiments or aspects, the aggregation learning system 102 may add a time delay amount to one or more data instances among a plurality of data instances of the training dataset, such that the time delay amount is input to the aggregation machine learning model for training and / or generating real-time output during runtime. In some non-limiting embodiments or aspects, the time delay amount may be an input feature. For example, the time delay amount may be added as an input feature to one or more data instances among a plurality of data instances to provide an augmented training dataset. Alternatively or additionally, the time delay amount may be added as an input feature to the aggregation machine learning model. In some non-limiting embodiments, the aggregation learning system 102 may provide the time delay amount as an input feature to the aggregation machine learning model. For example, the aggregation learning system 102 may provide the time delay amount as an input feature to the aggregation machine learning model during the training process. In some non-limiting embodiments, the aggregation machine learning model may be trained using the time delay amount as a direct model input variable.

[0142] Although the above systems, methods, and computer program products have been described in detail for illustrative purposes based on embodiments currently considered to be most practical and preferred, it should be understood that such details are for the purposes described only, and this disclosure is not limited to the described embodiments. Rather, this disclosure is intended to cover modifications and equivalent arrangements that fall within the spirit and scope of the appended claims. For example, it should be understood that this disclosure contemplates, as far as possible, that one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A system for aggregating learning using machine learning models to prevent model performance degradation, the system comprising: At least one processor, which is programmed or configured to: The system receives a training dataset of multiple data instances from a data source, wherein the training dataset includes a predefined static training dataset and a real-time training dataset, the real-time training dataset including new data generated in real time from an external component, each data instance including a time series of data points, and each data instance including transaction parameters associated with an electronic payment transaction, wherein the transaction parameters include at least one of the following: e-wallet card data associated with an e-card, decision data associated with a decision to approve or reject a transaction authorization request, authorization data associated with an authorization response, master account PAN, authorization code, data associated with transaction amount, data associated with transaction date and time, data associated with currency exchange rate, data associated with merchant type, data associated with the acquiring institution's country, data associated with an identifier of the country associated with the PAN, data associated with response code, data associated with merchant identifier, or data associated with the currency type corresponding to funds stored in association with the PAN. A time delay is added to one or more of the plurality of data instances to provide an augmented training dataset, wherein when the time delay is added to one or more of the plurality of data instances to provide the augmented training dataset, the at least one processor is further programmed or configured to: Add the time delay amount to multiple labeled data instances to provide multiple augmented labeled data instances; and The time delay is added to multiple unlabeled data instances to provide multiple augmented unlabeled data instances; and The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances; Selecting a first plurality of supervised machine learning models, the first plurality of supervised machine learning models comprising a plurality of neural network machine learning models trained using supervised learning techniques, and wherein when the first plurality of supervised machine learning models are selected, the at least one processor is programmed or configured to: Based on multiple augmented labeled data instances in the augmented training dataset, a sensitivity metric for time delay is determined for each supervised machine learning model in a set of supervised machine learning models, wherein the sensitivity metric for time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset compared to the augmented training dataset; and A subset of the set of supervised machine learning models is selected based on the sensitivity metric of each supervised machine learning model to time delay, wherein the first plurality of supervised machine learning models includes two or more machine learning models; Selecting a second or more unsupervised machine learning models, wherein the selection of the second or more unsupervised machine learning models includes: Based on the plurality of augmented unlabeled data instances in the augmented training dataset, a sensitivity measure to time delay is determined for each of a set of unsupervised machine learning models, wherein the sensitivity measure to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset; and A subset of the set of unsupervised machine learning models is selected based on the sensitivity metric of each unsupervised machine learning model to time delay, wherein the second plurality of unsupervised machine learning models includes two or more machine learning models; The first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models are trained based on the augmented training dataset; An aggregated machine learning model is generated based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and Receive transaction parameters associated with electronic payment transactions; The transaction parameters are input into the aggregated machine learning model for classification, and the time spent clearing the electronic payment transaction is generated as the runtime output of the aggregated machine learning model based on the transaction parameters.

2. The system of claim 1, wherein the at least one processor is further programmed or configured to: The runtime output of the aggregated machine learning model is used to execute actions.

3. The system of claim 1, wherein when generating the aggregated machine learning model, the at least one processor is programmed or configured to: The output of the aggregated machine learning model is generated based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and The aggregate machine learning model is trained using backpropagation based on its output.

4. The system of claim 1, wherein the at least one processor is further programmed or configured to: Add the perturbation to one or more additional data instances of the plurality of data instances; and Add one or more additional data instances from the plurality of data instances to the augmented training dataset.

5. A method for using machine learning models to aggregate learning to prevent model performance degradation, the method comprising: The system utilizes at least one processor to receive a training dataset of multiple data instances from a data source. The training dataset includes a predefined static training dataset and a real-time training dataset, the real-time training dataset including new data generated in real-time from an external component. Each data instance includes a time series of data points, and each data instance includes transaction parameters associated with an electronic payment transaction. These transaction parameters include at least one of the following: e-wallet card data associated with an e-card; decision data associated with a decision to approve or reject a transaction authorization request; authorization data associated with an authorization response; master account (PAN); authorization code; data associated with the transaction amount; data associated with the transaction date and time; data associated with the currency exchange rate; data associated with the merchant type; data associated with the acquiring institution's country; data associated with an identifier of the country associated with the PAN; data associated with the response code; data associated with the merchant identifier; or data associated with the currency type corresponding to funds stored in association with the PAN. Using at least one processor to add a time delay amount to one or more of the plurality of data instances to provide an augmented training dataset, wherein adding the time delay amount to one or more of the plurality of data instances to provide the augmented training dataset includes: Add the time delay amount to multiple labeled data instances to provide multiple augmented labeled data instances; and The time delay is added to multiple unlabeled data instances to provide multiple augmented unlabeled data instances; and The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances; Using at least one processor, a first plurality of supervised machine learning models are selected, the first plurality of supervised machine learning models comprising a plurality of neural network machine learning models trained using supervised learning techniques, wherein selecting the first plurality of supervised machine learning models includes: Based on multiple augmented labeled data instances in the augmented training dataset, a sensitivity metric for time delay is determined for each supervised machine learning model in a set of supervised machine learning models, wherein the sensitivity metric for time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset compared to the augmented training dataset; and A subset of the set of supervised machine learning models is selected based on the sensitivity metric of each supervised machine learning model to time delay, wherein the first plurality of supervised machine learning models includes two or more machine learning models; Selecting a second plurality of unsupervised machine learning models using at least one processor, wherein selecting the second plurality of unsupervised machine learning models includes: Based on the plurality of augmented unlabeled data instances in the augmented training dataset, a sensitivity measure to time delay is determined for each of a set of unsupervised machine learning models, wherein the sensitivity measure to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset; and A subset of the set of unsupervised machine learning models is selected based on the sensitivity metric of each unsupervised machine learning model to time delay, wherein the second plurality of unsupervised machine learning models includes two or more machine learning models; The first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models are trained using at least one processor based on the augmented training dataset; An aggregated machine learning model is generated using at least one processor based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and Receive transaction parameters associated with electronic payment transactions using at least one processor; The transaction parameters are input into the aggregated machine learning model using at least one processor for classification, so as to generate the amount of time spent clearing the electronic payment transaction as the runtime output of the aggregated machine learning model based on the transaction parameters.

6. The method according to claim 5, further comprising: The runtime output of the aggregated machine learning model is used to execute actions.

7. The method of claim 5, wherein generating the aggregated machine learning model comprises: The output of the aggregated machine learning model is generated based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; as well as The aggregate machine learning model is trained using backpropagation based on its output.

8. The method according to claim 5, further comprising: Add the perturbation to one or more additional data instances of the plurality of data instances; as well as Add one or more additional data instances from the plurality of data instances to the augmented training dataset.

9. A computer program product for aggregating learning using a machine learning model to prevent model performance degradation, the computer program product comprising at least one non-transient computer-readable medium, the at least one non-transient computer-readable medium comprising one or more instructions, the one or more instructions causing the at least one processor, when executed by at least one processor, to: Receive a training dataset from multiple data instances from a data source, where, The training dataset includes a predefined static training dataset and a real-time training dataset. The real-time training dataset includes new data generated in real time from external components. Each data instance includes a time series of data points. Each data instance includes transaction parameters associated with electronic payment transactions. The transaction parameters include at least one of the following: e-wallet card data associated with an e-card, decision data associated with the decision to approve or reject a transaction authorization request, authorization data associated with the authorization response, master account PAN, authorization code, data associated with the transaction amount, data associated with the transaction date and time, data associated with the currency exchange rate, data associated with the merchant type, data associated with the acquiring institution's country, data associated with the identifier of the country associated with the PAN, data associated with the response code, data associated with the merchant identifier, or data associated with the currency type corresponding to the funds stored in association with the PAN. Adding a time delay to one or more of the plurality of data instances to provide an augmented training dataset, wherein adding the time delay to one or more of the plurality of data instances to provide the augmented training dataset includes: Add the time delay amount to multiple labeled data instances to provide multiple augmented labeled data instances; and The time delay is added to multiple unlabeled data instances to provide multiple augmented unlabeled data instances; and The augmented training dataset includes the plurality of augmented labeled data instances and the plurality of augmented unlabeled data instances; Selecting a first plurality of supervised machine learning models, the first plurality of supervised machine learning models comprising a plurality of neural network machine learning models trained using supervised learning techniques, and wherein the at least one processor selects one or more instructions of the first plurality of supervised machine learning models, such that the at least one processor: Based on multiple augmented labeled data instances in the augmented training dataset, a sensitivity metric for time delay is determined for each supervised machine learning model in a set of supervised machine learning models, wherein the sensitivity metric for time delay includes a measure of the change in accuracy of the supervised machine learning model relative to the training dataset compared to the augmented training dataset; and A subset of the set of supervised machine learning models is selected based on the sensitivity metric of each supervised machine learning model to time delay, wherein the first plurality of supervised machine learning models includes two or more machine learning models; Selecting a second plurality of unsupervised machine learning models, wherein the at least one processor selects one or more instructions of the second plurality of unsupervised machine learning models, such that the at least one processor: Based on the plurality of augmented unlabeled data instances in the augmented training dataset, a sensitivity measure to time delay is determined for each of a set of unsupervised machine learning models, wherein the sensitivity measure to time delay includes a measure of the change in accuracy of the unsupervised machine learning model relative to the training dataset and relative to the augmented training dataset; and A subset of the set of unsupervised machine learning models is selected based on the sensitivity metric of each unsupervised machine learning model to time delay, wherein the second plurality of unsupervised machine learning models includes two or more machine learning models; The first plurality of supervised machine learning models and the second plurality of unsupervised machine learning models are trained based on the augmented training dataset; An aggregated machine learning model is generated based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and Receive transaction parameters associated with electronic payment transactions; The transaction parameters are input into the aggregated machine learning model for classification, and the time spent clearing the electronic payment transaction is generated as the runtime output of the aggregated machine learning model based on the transaction parameters.

10. The computer program product of claim 9, wherein the one or more instructions further cause the at least one processor to: The runtime output of the aggregated machine learning model is used to execute actions.

11. The computer program product of claim 9, wherein the one or more instructions that cause the at least one processor to generate the aggregated machine learning model cause the at least one processor to: The output of the aggregated machine learning model is generated based on the outputs of the first plurality of supervised machine learning models and the outputs of the second plurality of unsupervised machine learning models; and The aggregate machine learning model is trained using backpropagation based on its output.

12. The computer program product of claim 9, wherein the one or more instructions further cause the at least one processor to: Add the perturbation to one or more additional data instances of the plurality of data instances; and Add one or more additional data instances from the plurality of data instances to the augmented training dataset.

Images