System, method, and computer program product for enhancing machine learning model performance without altering the model or acquiring additional data

Abstract

Systems, methods, and computer program products are provided for enhancing machine learning model performance without altering the model or acquiring additional data. An example method includes inputting a data set to a first machine learning model to generate predicted labels. Incorrectly labeled data from the data set is identified based on comparing the predicted labels with known labels. A second machine learning model is trained based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy. Synthetic data is generated based on the data set. The first machine learning model is refined based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

Description

Attorney Docket No.08223-2500316 (8143WO01) SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR ENHANCING MACHINE LEARNING MODEL PERFORMANCE WITHOUT ALTERING THE MODEL OR ACQUIRING ADDITIONAL DATA CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to United States Provisional Patent Application No. 63 / 642,213, filed May 3, 2024, the disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND 1. Technical Field

[0002] This disclosure relates generally to machine learning and, in some non- limiting embodiments or aspects, to systems, methods, and computer program products for enhancing machine learning model performance without altering the model or acquiring additional data. 2. Technical Considerations

[0003] Machine learning models are successful in performing many real-world tasks. When a machine learning model makes a decision, one of two scenarios may happen: the model makes a correct decision or the model makes an incorrect decision. For example, a correct decision is when the model prediction matches or is identical to true value (e.g., a known or given label for a sample). An incorrect decision is when the model prediction is different than the true value (e.g., the known or given label for the sample).

[0004] If the model is well-designed and well-trained, the model will make a correct decision on a majority of the samples. However, there will still be incorrect decisions for some samples, prompting designers to improve performance of the model to minimize any incorrect decisions.

[0005] Acquiring additional data and / or using a modified model may improve performance. However, modifying the model introduces additional complexity to the model because changing a model is complicated after a model has been deployed, and obtaining additional data is not always an option (e.g., additional data may not be available and / or may be difficult and / or time-consuming to obtain).

[0006] There is a need for systems, methods, and products that can improve machine learning model performance using the same model and without obtaining new data. 63T8613.DOCX Page 1 of 55Attorney Docket No.08223-2500316 (8143WO01) SUMMARY

[0007] Accordingly, provided are improved systems, methods, and computer program products for enhancing machine learning model performance without altering the model or acquiring additional data.

[0008] According to non-limiting embodiments or aspects, provided is a computer- implemented method for enhancing machine learning model performance without altering the model or acquiring additional data. An example method may include inputting a data set to a first machine learning model to generate predicted labels. Incorrectly labeled data from the data set may be identified based on comparing the predicted labels with known labels for the data set. The predicted labels associated with the incorrectly labeled data may be different than the known labels. A second machine learning model may be trained based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data. Synthetic data may be generated based on the data set. The first machine learning model may be refined based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0009] In some non-limiting embodiments or aspects, the first machine learning model may include a classifier model.

[0010] In some non-limiting embodiments or aspects, the second machine learning model may include at least one of an evaluator model, a scoring model, or any combination thereof.

[0011] In some non-limiting embodiments or aspects, the score associated with accuracy of each predicted label associated with the incorrectly labeled data may include a first score. Training may include training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data. 63T8613.DOCX Page 2 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0012] In some non-limiting embodiments or aspects, the method may further include inputting the synthetic data to the first machine learning model to generate the synthetic predicted labels.

[0013] In some non-limiting embodiments or aspects, the refining may include initializing a refined model based on the first machine learning model and training the refined machine learning model.

[0014] In some non-limiting embodiments or aspects, the scores associated with the synthetic predicted labels may include first scores. The synthetic predicted labels associated with the synthetic data generated by the first machine learning model may include first synthetic predicted labels. Training the refined machine learning model may include inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels, inputting the synthetic data to the refined machine learning model to generate second synthetic predicted labels, inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels, and updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0015] In some non-limiting embodiments or aspects, the synthetic data may include a plurality of synthetic data samples. Generating the synthetic data may include generating each synthetic data sample of the plurality of synthetic data samples by selecting a sample from the data set, selecting a feature of the sample, sampling a synthetic value for the feature based on a distribution associated with the feature, and replacing a current value of the feature with the synthetic value for the feature in the sample. The synthetic data sample may include the sample with the synthetic value.

[0016] In some non-limiting embodiments or aspects, at least one feature of the sample may include a numeric feature. The distribution associated with each numeric feature may be associated with a mean value and a deviation value.

[0017] In some non-limiting embodiments or aspects, at least one feature of the sample may include a nominal feature. The distribution associated with each nominal feature may be associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof. 63T8613.DOCX Page 3 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0018] According to non-limiting embodiments or aspects, provided is a system for enhancing machine learning model performance without altering the model or acquiring additional data. An example system may include at least one processor, which may be configured to input a data set to a first machine learning model to generate predicted labels. The at least one processor may be configured to identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set. The predicted labels associated with the incorrectly labeled data may be different than the known labels. The at least one processor may be configured to train a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data. The at least one processor may be configured to generate synthetic data based on the data set. The at least one processor may be configured to refine the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0019] In some non-limiting embodiments or aspects, the first machine learning model may include a classifier model.

[0020] In some non-limiting embodiments or aspects, the second machine learning model may include at least one of an evaluator model, a scoring model, or any combination thereof.

[0021] In some non-limiting embodiments or aspects, the score associated with accuracy of each predicted label associated with the incorrectly labeled data may include a first score. Training may include training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

[0022] In some non-limiting embodiments or aspects, the at least one processor may be further configured to input the synthetic data to the first machine learning model to generate the synthetic predicted labels. 63T8613.DOCX Page 4 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0023] In some non-limiting embodiments or aspects, refining may include initializing a refined model based on the first machine learning model and training the refined machine learning model.

[0024] In some non-limiting embodiments or aspects, the scores associated with the synthetic predicted labels may include first scores, and the synthetic predicted labels associated with the synthetic data generated by the first machine learning model may include first synthetic predicted labels. Training the refined machine learning model may include inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels, inputting the synthetic data to the refined machine learning model to generate second synthetic predicted labels, inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels, and updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0025] In some non-limiting embodiments or aspects, the synthetic data may include a plurality of synthetic data samples. Generating the synthetic data may include generating each synthetic data sample of the plurality of synthetic data samples by selecting a sample from the data set, selecting a feature of the sample, sampling a synthetic value for the feature based on a distribution associated with the feature, and replacing a current value of the feature with the synthetic value for the feature in the sample. The synthetic data sample may include the sample with the synthetic value.

[0026] In some non-limiting embodiments or aspects, at least one feature of the sample may include a numeric feature. The distribution associated with each numeric feature may be associated with a mean value and a deviation value.

[0027] In some non-limiting embodiments or aspects, at least one feature of the sample may include a nominal feature. The distribution associated with each nominal feature may be associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

[0028] According to non-limiting embodiments or aspects, provided is a computer program product for enhancing machine learning model performance without altering the model or acquiring additional data. An example computer program product may include at least one non-transitory computer-readable medium including program 63T8613.DOCX Page 5 of 55Attorney Docket No.08223-2500316 (8143WO01) instructions that, when executed by at least one processor, cause the at least one processor to input a data set to a first machine learning model to generate predicted labels. The instructions, when executed by the at least one processor, may cause the at least one processor to identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set. The predicted labels associated with the incorrectly labeled data may be different than the known labels. The instructions, when executed by the at least one processor, may cause the at least one processor to train a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data. The instructions, when executed by the at least one processor, may cause the at least one processor to generate synthetic data based on the data set. The instructions, when executed by the at least one processor, may cause the at least one processor to refine the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0029] In some non-limiting embodiments or aspects, the first machine learning model may include a classifier model.

[0030] In some non-limiting embodiments or aspects, the second machine learning model may include at least one of an evaluator model, a scoring model, or any combination thereof.

[0031] In some non-limiting embodiments or aspects, the score associated with accuracy of each predicted label associated with the incorrectly labeled data may include a first score. Training may include training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

[0032] In some non-limiting embodiments or aspects, the instructions, when executed by the at least one processor, may cause the at least one processor to input 63T8613.DOCX Page 6 of 55Attorney Docket No.08223-2500316 (8143WO01) the synthetic data to the first machine learning model to generate the synthetic predicted labels.

[0033] In some non-limiting embodiments or aspects, refining may include initializing a refined model based on the first machine learning model and training the refined machine learning model.

[0034] In some non-limiting embodiments or aspects, the scores associated with the synthetic predicted labels may include first scores, and the synthetic predicted labels associated with the synthetic data generated by the first machine learning model may include first synthetic predicted labels. Training the refined machine learning model may include inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels, inputting the synthetic data to the refined machine learning model to generate second synthetic predicted labels, inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels, and updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0035] In some non-limiting embodiments or aspects, the synthetic data may include a plurality of synthetic data samples. Generating the synthetic data may include generating each synthetic data sample of the plurality of synthetic data samples by selecting a sample from the data set, selecting a feature of the sample, sampling a synthetic value for the feature based on a distribution associated with the feature, and replacing a current value of the feature with the synthetic value for the feature in the sample. The synthetic data sample may include the sample with the synthetic value.

[0036] In some non-limiting embodiments or aspects, at least one feature of the sample may include a numeric feature. The distribution associated with each numeric feature may be associated with a mean value and a deviation value.

[0037] In some non-limiting embodiments or aspects, at least one feature of the sample may include a nominal feature. The distribution associated with each nominal feature may be associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

[0038] Further non-limiting embodiments or aspects are set forth in the following numbered clauses: 63T8613.DOCX Page 7 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0039] Clause 1: A computer-implemented method, comprising: inputting, with at least one processor, a data set to a first machine learning model to generate predicted labels; identifying, with at least one processor, incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; training, with at least one processor, a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generating, with at least one processor, synthetic data based on the data set; and refining, with at least one processor, the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0040] Clause 2: The method of clause 1, wherein the first machine learning model comprises a classifier model.

[0041] Clause 3: The method of clause 1 or 2, wherein the second machine learning model comprises at least one of an evaluator model, a scoring model, or any combination thereof.

[0042] Clause 4: The method of any of clauses 1-3, wherein the score associated with accuracy of each predicted label associated with the incorrectly labeled data comprises a first score, and wherein training comprises training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

[0043] Clause 5: The method of any of clauses 1-4, further comprising: inputting, with at least one processor, the synthetic data to the first machine learning model to generate the synthetic predicted labels.

[0044] Clause 6: The method of any of clauses 1-5, wherein the refining comprises: initializing a refined model based on the first machine learning model; and training the refined machine learning model. 63T8613.DOCX Page 8 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0045] Clause 7: The method of any of clauses 1-6, wherein the scores associated with the synthetic predicted labels comprise first scores, wherein the synthetic predicted labels associated with the synthetic data generated by the first machine learning model comprise first synthetic predicted labels, and wherein training the refined machine learning model comprises: inputting, with at least one processor, the synthetic data to the first machine learning model to generate the first synthetic predicted labels; inputting, with at least one processor, the synthetic data to the refined machine learning model to generate second synthetic predicted labels; inputting, with at least one processor, the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels; and updating, with at least one processor, parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0046] Clause 8: The method of any of clauses 1-7, wherein the synthetic data comprises a plurality of synthetic data samples, and wherein generating the synthetic data comprises generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set; selecting a feature of the sample; sampling a synthetic value for the feature based on a distribution associated with the feature; and replacing a current value of the feature with the synthetic value for the feature in the sample, wherein the synthetic data sample comprises the sample with the synthetic value.

[0047] Clause 9: The method of any of clauses 1-8, wherein at least one feature of the sample comprises a numeric feature, and wherein the distribution associated with each numeric feature is associated with a mean value and a deviation value.

[0048] Clause 10: The method of any of clauses 1-9, wherein at least one feature of the sample comprises a nominal feature, and wherein the distribution associated with each nominal feature is associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

[0049] Clause 11: A system comprising at least one processor configured to: input a data set to a first machine learning model to generate predicted labels; identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; train a second machine 63T8613.DOCX Page 9 of 55Attorney Docket No.08223-2500316 (8143WO01) learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generate synthetic data based on the data set; and refine the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0050] Clause 12: The system of clause 11, wherein the first machine learning model comprises a classifier model.

[0051] Clause 13: The system of clause 11 or 12, wherein the second machine learning model comprises at least one of an evaluator model, a scoring model, or any combination thereof.

[0052] Clause 14: The system of any of clauses 11-13, wherein the score associated with accuracy of each predicted label associated with the incorrectly labeled data comprises a first score, and wherein training comprises training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

[0053] Clause 15: The system of any of clauses 11-14, wherein the at least one processor is further configured to: input the synthetic data to the first machine learning model to generate the synthetic predicted labels.

[0054] Clause 16: The system of any of clauses 11-15, wherein refining comprises: initializing a refined model based on the first machine learning model; and training the refined machine learning model.

[0055] Clause 17: The system of any of clauses 11-16, wherein the scores associated with the synthetic predicted labels comprise first scores, wherein the synthetic predicted labels associated with the synthetic data generated by the first machine learning model comprise first synthetic predicted labels, and wherein training the refined machine learning model comprises: inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels; inputting the 63T8613.DOCX Page 10 of 55Attorney Docket No.08223-2500316 (8143WO01) synthetic data to the refined machine learning model to generate second synthetic predicted labels; inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels; and updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0056] Clause 18: The system of any of clauses 11-17, wherein the synthetic data comprises a plurality of synthetic data samples, and wherein generating the synthetic data comprises generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set; selecting a feature of the sample; sampling a synthetic value for the feature based on a distribution associated with the feature; and replacing a current value of the feature with the synthetic value for the feature in the sample, wherein the synthetic data sample comprises the sample with the synthetic value.

[0057] Clause 19: The system of any of clauses 11-18, wherein at least one feature of the sample comprises a numeric feature, and wherein the distribution associated with each numeric feature is associated with a mean value and a deviation value.

[0058] Clause 20: The system of any of clauses 11-19, wherein at least one feature of the sample comprises a nominal feature, and wherein the distribution associated with each nominal feature is associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

[0059] Clause 21: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: input a data set to a first machine learning model to generate predicted labels; identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; train a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generate synthetic data based on the data set; and refine the first machine learning model based on the synthetic data, synthetic predicted 63T8613.DOCX Page 11 of 55Attorney Docket No.08223-2500316 (8143WO01) labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

[0060] Clause 22: The computer program product of clause 21, wherein the first machine learning model comprises a classifier model.

[0061] Clause 23: The computer program product of clause 21 or 22, wherein the second machine learning model comprises at least one of an evaluator model, a scoring model, or any combination thereof.

[0062] Clause 24: The computer program product of any of clauses 21-23, wherein the score associated with accuracy of each predicted label associated with the incorrectly labeled data comprises a first score, and wherein training comprises training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

[0063] Clause 25: The computer program product of any of clauses 21-24, wherein the at least one processor is further configured to: input the synthetic data to the first machine learning model to generate the synthetic predicted labels.

[0064] Clause 26: The computer program product of any of clauses 21-25, wherein refining comprises: initializing a refined model based on the first machine learning model; and training the refined machine learning model.

[0065] Clause 27: The computer program product of any of clauses 21-26, wherein the scores associated with the synthetic predicted labels comprise first scores, wherein the synthetic predicted labels associated with the synthetic data generated by the first machine learning model comprise first synthetic predicted labels, and wherein training the refined machine learning model comprises: inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels; inputting the synthetic data to the refined machine learning model to generate second synthetic predicted labels; inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels; and 63T8613.DOCX Page 12 of 55Attorney Docket No.08223-2500316 (8143WO01) updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

[0066] Clause 28: The computer program product of any of clauses 21-27, wherein the synthetic data comprises a plurality of synthetic data samples, and wherein generating the synthetic data comprises generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set; selecting a feature of the sample; sampling a synthetic value for the feature based on a distribution associated with the feature; and replacing a current value of the feature with the synthetic value for the feature in the sample, wherein the synthetic data sample comprises the sample with the synthetic value.

[0067] Clause 29: The computer program product of any of clauses 21-28, wherein at least one feature of the sample comprises a numeric feature, and wherein the distribution associated with each numeric feature is associated with a mean value and a deviation value.

[0068] Clause 30: The computer program product of any of clauses 21-29, wherein at least one feature of the sample comprises a nominal feature, and wherein the distribution associated with each nominal feature is associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

[0069] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0071] FIG.1 is a schematic diagram of an example system for enhancing machine learning model performance without altering the model or acquiring additional data, according to some non-limiting embodiments or aspects; 63T8613.DOCX Page 13 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0072] FIG. 2 is a flow diagram for an example method for enhancing machine learning model performance without altering the model or acquiring additional data, according to some non-limiting embodiments or aspects;

[0073] FIG. 3 is a diagram of an example payment processing network in which systems, methods, and / or computer program products, described herein, may be implemented, according to some non-limiting embodiments or aspects;

[0074] FIG. 4 is a schematic diagram of example components of one or more devices of FIG. 1 and / or FIG. 3, according to some non-limiting embodiments or aspects; and

[0075] FIGS. 5A-5D are schematic diagrams of an example implementation of systems and methods for enhancing machine learning model performance without altering the model or acquiring additional data, according to some non-limiting embodiments or aspects. DETAILED DESCRIPTION

[0076] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0077] Some non-limiting embodiments or aspects may be described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0078] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and 63T8613.DOCX Page 14 of 55Attorney Docket No.08223-2500316 (8143WO01) “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).

[0079] As used herein, the term “account identifier” may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” may refer to an identifier that is used as a substitute or replacement identifier for an original account identifier, such as a PAN. Account identifiers may be alphanumeric or any combination of characters and / or symbols. Tokens may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, and / or the like) such that they may be used to conduct a transaction without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with a plurality of tokens for different individuals or purposes.

[0080] As used herein, the term “acquirer institution” may refer to an entity licensed and / or approved by a transaction service provider to originate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. The transactions the acquirer institution may originate may include payment transactions (e.g., purchases, original credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, an acquirer institution may be a financial institution, such as a bank. As used herein, the term “acquirer system” may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0081] As used herein, the terms “client” and “client device” may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider) 63T8613.DOCX Page 15 of 55Attorney Docket No.08223-2500316 (8143WO01) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a “client device” may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and / or the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate transactions. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, and / or the like), personal digital assistants (PDAs), and / or the like. Moreover, a “client” may also refer to an entity (e.g., a merchant, an acquirer, and / or the like) that owns, utilizes, and / or operates a client device for initiating transactions (e.g., for initiating transactions with a transaction service provider).

[0082] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0083] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, 63T8613.DOCX Page 16 of 55Attorney Docket No.08223-2500316 (8143WO01) such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a PDA, and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0084] As used herein, the terms “electronic wallet” and “electronic wallet application” refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device executing an electronic wallet application, and may further include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An “electronic wallet provider” may include an entity that provides and / or maintains an electronic wallet for a customer, such as Google Pay®, Android Pay®, Apple Pay®, Samsung Pay®, and / or other like electronic payment systems. In some non-limiting examples, an issuer bank may be an electronic wallet provider.

[0085] As used herein, the term “issuer institution” may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide an account identifier, such as a PAN, to a customer that uniquely identifies one or more accounts associated with that customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument, e.g., a payment card, and / or may be electronic and used for electronic payments. The term “issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing a transaction.

[0086] As used herein, the term “merchant” may refer to an individual or entity that provides goods and / or services, or access to goods and / or services, to customers based on a transaction, such as a payment transaction. The term “merchant” or “merchant system” may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer executing one or more software applications. 63T8613.DOCX Page 17 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0087] As used herein, the term “payment device” may refer to an electronic payment device, a portable financial device, a payment card (e.g., a credit or debit card), a gift card, a smartcard, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, a radio frequency identification (RFID) transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, and / or the like. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, a name of the account holder, and / or the like).

[0088] As used herein, the term “payment gateway” may refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, a payment service provider, a payment facilitator, a payment facilitator that contracts with an acquirer, a payment aggregator, and / or the like), which provides payment services (e.g., transaction service provider payment services, payment processing services, and / or the like) to one or more merchants. The payment services may be associated with the use of portable financial devices managed by a transaction service provider. As used herein, the term “payment gateway system” may refer to one or more computer systems, computer devices, servers, groups of servers, and / or the like, operated by or on behalf of a payment gateway.

[0089] As used herein, a “point-of-sale (POS) device” may refer to one or more devices, which may be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include peripheral devices, card readers, scanning devices (e.g., code scanners), Bluetooth® communication receivers, near-field communication (NFC) receivers, RFID receivers, and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, and / or the like. As used herein, a “point-of-sale (POS) system” may refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system may include one or more POS devices and / or other like devices that may be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) 63T8613.DOCX Page 18 of 55Attorney Docket No.08223-2500316 (8143WO01) may include one or more server computers configured to process online payment transactions through webpages, mobile applications, and / or the like.

[0090] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, POS devices, mobile devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0091] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing a first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0092] As used herein, the term “transaction service provider” may refer to an entity that receives transaction authorization requests from merchants or other entities and provides guarantees of payment, in some cases through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider may include a payment network such as Visa® or any other entity that processes transactions. The term “transaction processing system” may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0093] Non-limiting embodiments or aspects of the disclosed subject matter are directed to systems, methods, and computer program products for enhancing machine learning model performance without altering the model or acquiring additional data. For example, a first machine learning model is used to generate predicted labels for a 63T8613.DOCX Page 19 of 55Attorney Docket No.08223-2500316 (8143WO01) data set. The incorrectly labeled data is identified and then used to train a second machine learning model, which generates a score associated with accuracy of the predicted labels of the incorrectly labeled data. Synthetic data is generated using the initial data set and provided back to the first machine learning model, which is refined based on scores generated by the second machine learning model associated with accuracy of synthetic predicted labels generated by the first machine learning model. As such, the present disclosure allows for an improved model performance using the same data and the same model (i.e., without altering the model or acquiring additional data). For example, non-limiting embodiments or aspects of the disclosed subject matter provide training a second machine learning model as an evaluator on samples which the first machine model labels incorrectly. The second machine learning model may receive data and a prediction (e.g., label or model prediction) and output a score indicating the quality of the prediction. This score associated with accuracy may be used to assist in improving the machine learning model as part of a feedback loop. Additionally, the disclosed subject matter provides flexibility and allows for any machine learning models to be used (e.g., there is no limitation to a particular model or type of model). As such, the disclosed techniques may be implemented in countless environments (e.g., any suitable environment in which a machine learning model may be useful). Moreover, the disclosed subject matter provides for using incorrectly labeled data to improve performance and accurately assess the performance of the machine learning model. For example, a higher percentage of incorrectly labeled data generally correlates to a less accurate score associated with the machine learning model. Furthermore, non-limiting embodiments or aspects of the disclosed subject matter provide implementing a synthetic data generator to generate synthetic data. The synthetic data may be generated based on an already present initial data set. For example, non-limiting embodiments or aspects of the disclosed subject matter may utilize a statistics extractor to determine the distribution of the initial data set. A data generator may, thereafter, be used to generate additional data that falls within or has a similar distribution to the initial data set. Because the initial data set may include numeric and / or nominal features, the synthetic data may also include numeric and / or nominal features. As such, a large amount of unseen data with a similar distribution to already existing data may be generated and used to improve and train the machine learning model. In addition, being able to include this additional (generated) data to train and refine the machine learning model promotes a more robust model and allows 63T8613.DOCX Page 20 of 55Attorney Docket No.08223-2500316 (8143WO01) for the model to be further refined to improve accuracy. For example, non-limiting embodiments or aspects of the disclosed subject matter may utilize a loss function that compares the outputs of the first machine learning model with the outputs of a refined machine learning model, where the minimized loss correlates to instances when the machine learning model performs similarly to the refined machine learning model.

[0094] Referring now to FIG. 1, shown is example system 100 for enhancing machine learning models, according to some non-limiting embodiments or aspects. For example, system 100 may include machine learning model system 102, database 108, user device 110, and / or communication network 112.

[0095] Machine learning model system 102 may include one or more devices capable of receiving information from and / or communicating information to database 108 and / or user device 110 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, machine learning model system 102 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, machine learning model system 102 may include a data storage device (e.g., database 108). In some non-limiting embodiments or aspects, machine learning model system 102 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device (e.g., database 108).

[0096] In some non-limiting embodiments or aspects, machine learning model system 102 may include first model 104 and / or second model 106. In some non- limiting embodiments or aspects, first model 104 and / or second model 106 each may include one or more machine learning models. In some non-limiting embodiments or aspects, second model 106 may include one or more machine learning models which may be trained and subsequently used to refine first model 104 using various techniques, such as reinforcement learning with self-critic and regularization, and / or the like. Second model 106 may be used to train first model 104 to make the same predictions as second model 106.

[0097] In some non-limiting embodiments or aspects, machine learning model system 102 may generate (e.g., train, validate, retrain, and / or the like), store, and / or implement (e.g., operate, provide inputs to and / or outputs from, and / or the like) one or 63T8613.DOCX Page 21 of 55Attorney Docket No.08223-2500316 (8143WO01) more machine learning models, such as first model 104 and / or second model 106. For example, machine learning model system 102 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices.

[0098] In some non-limiting embodiments or aspects, machine learning model system 102 may be in communication with a data storage device, which may be local or remote to machine learning model system 102. In some non-limiting embodiments or aspects, machine learning model system 102 may be capable of receiving information from, storing information in, transmitting information to, and / or searching information stored in database 108 and / or user device 110 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like).

[0099] Database 108 may include one or more devices capable of receiving information from and / or communicating information to machine learning model system 102 and / or user device 110 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, database 108 may include a computing device, such as a server, a group of servers, a desktop computer, a portable computer, a mobile device, and / or other like devices. In some non-limiting embodiments or aspects, database 108 may include a data storage device. In some non-limiting embodiments or aspects, database 108 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, database 108 may be part of machine learning model system 102 and / or part of the same system as machine learning model system 102.

[0100] User device 110 may include one or more devices capable of receiving information from and / or communicating information to machine learning model system 102 and / or database 108 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or the like). For example, user device 110 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other like devices. Additionally or alternatively, each user device 110 may include a device capable of receiving information from and / or communicating information to other user devices 110 (e.g., directly via wired or wireless communication connection, indirectly via communication network 112, and / or 63T8613.DOCX Page 22 of 55Attorney Docket No.08223-2500316 (8143WO01) the like). In some non-limiting embodiments or aspects, user device 110 may be part of machine learning model system 102 and / or part of the same system as machine learning model system 102. For example, machine learning model system 102, database 108, and user device 110 may all be (and / or be part of) a single system and / or a single computing device.

[0101] Communication network 112 may include one or more wired and / or wireless networks. For example, communication network 112 may include a cellular network (e.g., a long-term evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0102] The number and arrangement of systems and devices shown in FIG.1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system or device, or a single system or device shown in FIG.1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another set of systems or another set of devices of system 100.

[0103] Referring now to FIG. 2, shown is a flow diagram for an example method 200 for enhancing machine learning models, according to some non-limiting embodiments or aspects. The steps shown in FIG.2 are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step. In some non-limiting embodiments or aspects, one or more of the steps of method 200 may be performed (e.g., completely, partially, etc.) by machine learning model system 102 (e.g., one or more devices of 63T8613.DOCX Page 23 of 55Attorney Docket No.08223-2500316 (8143WO01) machine learning model system 102). In some non-limiting embodiments or aspects, one or more of the steps of method 200 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including machine learning model system 102 (e.g., one or more devices of machine learning model system 102), such as database 108 (e.g., one or more devices of database 108) and / or user device 110.

[0104] As shown in FIG.2, at step 202, method 200 may include inputting a data set into a first machine learning model to generate predicted labels. For example, machine learning model system 102 may input a data set into a first machine learning model to generate predicted labels.

[0105] In some non-limiting embodiments or aspects, the data set may be a training data set. For example, the training data set may be received (e.g., retrieved, obtained, and / or the like) from database 108 (e.g., machine learning model system 102 may receive the training data set from database 108).

[0106] In some non-limiting embodiments or aspects, the training data set may include a plurality of training samples. In some non-limiting embodiments or aspects, each training sample of the plurality of the training samples of the training data set may be labeled.

[0107] In some non-limiting embodiments or aspects, the plurality of training samples may include a plurality of input embeddings (e.g., embedding vectors). In some non-limiting embodiments or aspects, the plurality of input embeddings may be word embeddings. In some non-limiting embodiments or aspects, the training data set may be stored in a storage component and / or stored in database 108.

[0108] In some non-limiting embodiments or aspects, the training data set may include data set biases.

[0109] In some non-limiting embodiments or aspects, machine learning model system 102 may provide the training data set as the input to first model 104. In some non-limiting embodiments or aspects, machine learning model system 102 may receive the training data set corresponding to an output from one or more machine learning models. In some non-limiting embodiments or aspects, machine learning model system 102 may input the training data set corresponding to the output from one or more machine learning models into another one or more machine learning models. For example, machine learning model system 102 may receive the training 63T8613.DOCX Page 24 of 55Attorney Docket No.08223-2500316 (8143WO01) data set corresponding to the output of a trained machine learning model, where the output may be provided as an input to first model 104 and / or second model 106.

[0110] In some non-limiting embodiments or aspects, each training sample of the plurality of training samples of the training data set may represent an institution (e.g., issuer, bank, merchant, and / or the like). In some non-limiting embodiments or aspects, each training sample of the plurality of training samples of the training data set may be associated with an event (e.g., transaction, account opening, fund transfer, etc.). In some non-limiting embodiments or aspects, each training sample of the plurality of training samples may be received in substantially real time with respect to an event (e.g., in real time, in near real time, during the event, as soon as practically available after the event, during processing and / or communication of messages related to the event, at the time of making a decision (e.g., authorization decision, authentication decision and / or the like) related to the event (e.g., receiving an authorization request, at least a portion of which is included in the data sample, and determining an authorization decision based thereon), and / or the like). In some non-limiting embodiments or aspects, each training sample of the plurality of training samples may include transaction data associated with an electronic payment transaction. In some non-limiting embodiments or aspects, transaction data may include transaction parameters associated with an electronic payment transaction. Transaction parameters may include electronic wallet card data associated with an electronic card (e.g., an electronic credit card, an electronic debit card, an electronic loyalty card, and / or the like), decision data associated with a decision (e.g., a decision to approve or deny a transaction authorization request), authorization data associated with an authorization response (e.g., an approved spending limit, an approved transaction value, and / or the like), a PAN, an authorization code (e.g., a personal identification number (PIN), etc.), data associated with a transaction amount (e.g., an approved limit, a transaction value, etc.), data associated with a transaction date and time, data associated with a conversion rate of a currency, data associated with a merchant type (e.g., goods, grocery, fuel, and / or the like), data associated with an acquiring institution country, data associated with an identifier of a country associated with the PAN, data associated with a response code, data associated with a merchant identifier (e.g., a merchant name, a merchant location, and / or the like), data associated with a type of currency corresponding to funds stored in association with the PAN, and / or the like. In 63T8613.DOCX Page 25 of 55Attorney Docket No.08223-2500316 (8143WO01) some non-limiting embodiments or aspects, each training sample of the plurality of training samples may be stored and compiled into a training data set for future training.

[0111] In some non-limiting embodiments or aspects, the data set may be input into a first machine learning model (e.g., first model 104) to generate predicted labels. For example, the predicted labels may serve to classify each sample in the data set. In some non-limiting embodiments or aspects, the predicted labels may be correct or incorrect. In some non-limiting embodiments or aspects, the number of correctly labeled data may exceed the number of incorrectly labeled data, such that the number of incorrectly labeled data makes up a small fraction of the overall labeled data. In some non-limiting embodiments or aspects, the first machine learning model may include a classifier model.

[0112] In some non-limiting embodiments or aspects, each training sample of the plurality of the training samples of the training data set may be labeled. For example, generating the predicted labels may provide (e.g., allow a user to have) two sets of labels for each data set: known labels and predicted labels.

[0113] As shown in FIG. 2, at step 204, method 200 may include identifying incorrectly labeled data from the data set. For example, machine learning model system 102 may identify incorrectly labeled data from the data set.

[0114] In some non-limiting embodiments or aspects, the incorrectly labeled data may be identified by comparing the predicted labels with known labels for the data set. When the predicted labels are different than the known labels, the predicted labels may be deemed as incorrect (e.g., incorrectly labeled data).

[0115] As shown in FIG.2, at step 206, method 200 may include training a second machine learning model using the incorrectly labeled data to generate accuracy scores. For example, machine learning model system 102 may train a second machine learning model using the incorrectly labeled data to generate accuracy scores.

[0116] In some non-limiting embodiments or aspects, the second machine learning model may be second model 106. In some non-limiting embodiments or aspects, the second machine learning model is an evaluator model, a scoring model, or any combination thereof.

[0117] In some non-limiting embodiments or aspects, the second machine learning model may be trained based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the 63T8613.DOCX Page 26 of 55Attorney Docket No.08223-2500316 (8143WO01) incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data. The score associated with accuracy of each predicted label associated with the incorrectly labeled data may be a first score. In some non-limiting embodiments or aspects, a second score associated with accuracy of each known label associated with the incorrectly labeled data may be generated.

[0118] In some non-limiting embodiments or aspects, the second machine learning model may use the first accuracy score for the predicted labels associated with the incorrectly labeled data and the second accuracy score for known labels associated with the incorrectly labeled data to determine loss according to a loss function. For example, the loss may be based on the difference between the first accuracy score for the predicted labels associated with the incorrectly labeled data and the second accuracy score for known labels. In some non-limiting embodiments or aspects, if the difference is negative (e.g., the second accuracy score for known labels is greater than the first accuracy score for the predicted labels associated with the incorrectly labeled data), the loss may be set to 0. In some non-limiting embodiments or aspects, the closer the accuracy scores are for the predicted labels and known labels, the lower the loss from the loss function. In some non-limiting embodiments or aspects, one or more parameters of the second machine learning model may be updated based on the determined loss.

[0119] As shown in FIG. 2, at step 208, method 200 may include generating synthetic data based on the data samples from the training data set. For example, machine learning model system 102 may generate synthetic data based on the data sample from the training set.

[0120] In some non-limiting embodiments or aspects, the synthetic data may be generated by a synthetic data generator. The synthetic data generator may include a statistics extractor and a data generator. As provided above, the synthetic data may be generated based on an already present initial data set. For example, the statistics extractor may determine the distribution of the initial or training data set, such as a mean value and deviation value (e.g., for numeric features), a binomial or multinomial distribution (e.g., for nominal features), among others.

[0121] The data generator may, thereafter, be used to generate additional data that falls within or has a similar distribution to the training data set. For example, in some non-limiting embodiments or aspects, generating the synthetic data includes 63T8613.DOCX Page 27 of 55Attorney Docket No.08223-2500316 (8143WO01) generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set, selecting a feature of the sample, sampling a synthetic value for the feature based on a distribution associated with the feature, and replacing a current value of the feature with the synthetic value for the feature in the sample. The synthetic data sample includes the sample with the synthetic value. The features may be numeric or nominal.

[0122] As shown in FIG. 2, at step 210, method 200 includes refining the first machine learning model with synthetic data and generated accuracy scores. For example, machine learning model system 102 may refine the first machine learning model with synthetic data and the generated accuracy scores.

[0123] Refining the first machine learning model may include: initializing a refined model based on the first machine learning module and training the refined machine learning model.

[0124] In some non-limiting embodiments or aspects, the first machine learning model is refined based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model. For example, in some non-limiting embodiments or aspects, the synthetic data may be input into the first machine learning model to generate synthetic predicted labels (e.g., first synthetic predicted labels).

[0125] In some non-limiting embodiments or aspects, training the refined machine learning model may include inputting the synthetic data into the refined machine learning model to generate second synthetic predicted labels.

[0126] In some non-limiting embodiments or aspects, the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels may be input into the second machine learning model to generate a first score associated with the accuracy of the first synthetic predicted labels and a second score associated with the accuracy of the second synthetic predicted labels.

[0127] In some non-limiting embodiments or aspects, the refined machine learning model may use the first accuracy score for the synthetic predicted labels and the second accuracy score for synthetic predicted labels to determine loss according to a loss function. One or more parameters of the refined machine learning model may then be updated based on the determined loss. 63T8613.DOCX Page 28 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0128] In some non-limiting embodiments or aspects, machine learning model system 102 may validate one or more machine learning models. For example, after training one or more machine learning models, machine learning model system 102 may validate one or more machine learning models. In some non-limiting embodiments or aspects, machine learning model system 102 may validate one or more machine learning models 104, 106 based on a validation threshold.

[0129] In some non-limiting embodiments or aspects, machine learning model system 102 may store one or more trained machine learning models. For example, machine learning model system 102 may store trained first model 104 and / or trained second model 106 in a data structure (e.g., database 108). The data structure may be located within machine learning model system 102 or external (e.g., remote from) machine learning model system 102.

[0130] Referring now to FIG. 3, depicted is a diagram of an example payment processing network 300, according to non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, payment processing network 300 may be used in conjunction with the systems, methods, and / or computer program products described herein, and / or the systems, methods, and / or computer program products described herein may be implemented in payment processing network 300. As shown in FIG. 3, payment processing network 300 may include transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310. In some non-limiting embodiments or aspects, each of machine learning model system 102, data storage system 108, and / or user device 110 of FIG. 1 may be implemented by (e.g., part of) transaction processing system 301. In some non-limiting embodiments or aspects, at least one of machine learning model system 102, database 108, and / or user device 110 of FIG.1 may be implemented by (e.g., part of) another system, another device, another group of systems, or another group of devices, separate from or including transaction processing system 301, such as merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like. For example, user device 110 may be implemented by (e.g., part of) at least one of payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310. Additionally or alternatively, for example, database 108 may be implemented by (e.g., part of) at least one of payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310. 63T8613.DOCX Page 29 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0131] Transaction processing system 301 may include one or more devices capable of receiving information from and / or communicating information to payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, transaction processing system 301 may be in communication with one or more issuer systems (e.g., issuer system 306), one or more acquirer systems (e.g., acquirer system 308), and / or one or more payment gateway systems (e.g., payment gateway system 302). Although only a single issuer system 306, single acquirer system 308, and single payment gateway system 302 are shown, it will be appreciated that transaction processing system 301 may be in communication with a plurality of issuer systems, a plurality of acquirer systems, and / or a plurality of payment gateway systems 302. In some non-limiting embodiments or aspects, transaction processing system 301 may include a computing device, such as a server (e.g., a transaction processing server), a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, transaction processing system 301 may be in communication with a data storage device, which may be local or remote to transaction processing system 301. In some non-limiting embodiments or aspects, transaction processing system 301 may be capable of receiving information from, storing information in, communicating information to, or searching information stored in the data storage device. In some non-limiting embodiments or aspects, transaction processing system 301 may be associated with a transaction service provider, as described herein. In some non-limiting embodiments or aspects, transaction processing system 301 may also operate as an issuer system, such that both transaction processing system 301 and issuer system 306 are a single system and / or controlled by a single entity.

[0132] Payment gateway system 302 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301, merchant system 304, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG.3, payment gateway system 302 may be in communication with one or more merchant systems (e.g., merchant system 304), one or more acquirer systems (e.g., acquirer system 308), and / or one or more transaction processing systems (e.g., 63T8613.DOCX Page 30 of 55Attorney Docket No.08223-2500316 (8143WO01) transaction processing system 301). Although only a single merchant system 304, single acquirer system 308, and single transaction processing system 301 are shown, it will be appreciated that payment gateway system 302 may be in communication with a plurality of merchant systems, a plurality of acquirer systems, and / or a plurality of transaction processing systems. In some non-limiting embodiments or aspects, payment gateway system 302 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non-limiting embodiments or aspects, payment gateway system 302 may be associated with a payment gateway, as described herein.

[0133] Merchant system 304 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301, payment gateway system 302, issuer system 306, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG.3, merchant system 304 may be in communication with one or more payment gateway systems (e.g., payment gateway system 302), one or more acquirer systems (e.g., acquirer system 308), and / or one or more consumer devices (e.g., consumer device 310). Although only a single payment gateway system 302, single acquirer system 308, and single consumer device 310 are shown, it will be appreciated that merchant system 304 may be in communication with a plurality of payment gateway systems, a plurality of acquirer systems, and / or a plurality of consumer devices. In some non-limiting embodiments or aspects, merchant system 304 may include a computing device, such as a server, a group of servers, a client device, a group of client devices, a POS device, a POS system, computers, computer systems, peripheral devices, and / or other like devices. In some non-limiting embodiments or aspects, merchant system 304 may be associated with a merchant, as described herein. In some non-limiting embodiments or aspects, merchant system 304 may include a device capable of receiving information from and / or communicating information to consumer device 310 via a short range communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or the like) with consumer device 310 and / or the like. In some non-limiting embodiments or aspects, merchant system 304 may include one or more client devices. For example, merchant system 304 may include a client device that allows a merchant to 63T8613.DOCX Page 31 of 55Attorney Docket No.08223-2500316 (8143WO01) communicate information to transaction processing system 301 (e.g., via at least one of acquirer system 308 and / or payment gateway system 302). In some non-limiting embodiments or aspects, merchant system 304 (e.g., a client device thereof, a POS device thereof, and / or the like) may also operate as a payment gateway system, such that both merchant system 304 and payment gateway system 302 are a single system and / or controlled by a single entity.

[0134] Issuer system 306 may include one or more devices capable of receiving information and / or communicating information to transaction processing system 301, payment gateway system 302, merchant system 304, acquirer system 308, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, issuer system 306 may be in communication with one or more transaction processing systems (e.g., transaction processing system 301) and / or one or more consumer devices (e.g., consumer device 310). Although only a single transaction processing system 301 and a single consumer device 310 are shown, it will be appreciated that issuer system 306 may be in communication with a plurality of transaction processing systems and / or a plurality of consumer devices 310. In some non-limiting embodiments or aspects, issuer system 306 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non- limiting embodiments or aspects, issuer system 306 may be associated with an issuer institution, as described herein. For example, issuer system 306 may be associated with an issuer institution that issued a credit account, debit account, credit card, debit card, a payment device, and / or the like to a user associated with consumer device 310.

[0135] Acquirer system 308 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, consumer device 310, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG.3, acquirer system 308 may be in communication with one or more transaction processing systems (e.g., transaction processing system 301), one or more payment gateway systems (e.g., payment gateway system 302), and / or one or more merchant systems (e.g., merchant system 304). Although only a single transaction processing system 301, a single payment gateway system 302, and a single merchant system 63T8613.DOCX Page 32 of 55Attorney Docket No.08223-2500316 (8143WO01) 304 are shown, it will be appreciated that acquirer system 308 may be in communication with a plurality of transaction processing systems, a plurality of payment gateway systems, and / or a plurality of merchant systems. In some non- limiting embodiments or aspects, acquirer system 308 may include a computing device, such as a server, a group of servers, and / or other like devices. In some non- limiting embodiments or aspects, acquirer system 308 may be associated with an acquirer institution, as described herein.

[0136] Consumer device 310 may include one or more devices capable of receiving information from and / or communicating information to transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or the like (e.g., directly, indirectly, via a public and / or private communication network connection, and / or the like). For example, as shown in FIG. 3, consumer device 310 may be in communication with one or more merchant systems (e.g., merchant system 304) and / or one or more issuer systems (e.g., issuer system 306). Although only a single merchant system 304 and a single issuer system 306 are shown, it will be appreciated that consumer device 310 may be in communication with a plurality of merchant systems and / or a plurality of issuer systems. In some non- limiting embodiments or aspects, consumer device 310 may be associated with a user to whom a credit account, a debit account, a credit card, a debit card, a payment device, and / or the like has been issued. In some non-limiting embodiments or aspects, consumer device 310 may include a computing device, such as a computer, a portable computer, a laptop computer, a tablet computer, a mobile device, a cellular phone, a smartphone, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a PDA, a client device, and / or other like devices. In some non-limiting embodiments or aspects, consumer device 310 may include a payment device, as described herein. In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of receiving information from and / or communicating information to other consumer devices 310 (e.g., directly, indirectly, via a public and / or private communication network connection, a short range communication connection, and / or the like). In some non-limiting embodiments or aspects, consumer device 310 may include a device capable of receiving information from and / or communicating information to merchant system 304 via a short range communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, and / or 63T8613.DOCX Page 33 of 55Attorney Docket No.08223-2500316 (8143WO01) the like) with merchant system 304 and / or the like. In some non-limiting embodiments or aspects, consumer device 310 may include a client device.

[0137] In some non-limiting embodiments or aspects, transaction processing system 301 may communicate with merchant system 304 directly (e.g., via a public and / or private communication network connection and / or the like). Additionally or alternatively, transaction processing system 301 may communicate with merchant system 304 through payment gateway system 302 and / or acquirer system 308. In some non-limiting embodiments or aspects, acquirer system 308 may be associated with merchant system 304 and / or may operate as payment gateway system 302 to facilitate the communication of transaction messages (e.g., authorization requests) from merchant system 304 to transaction processing system 301. In some non-limiting embodiments or aspects, merchant system 304 may communicate with payment gateway system 302 directly (e.g., via a public and / or private communication network connection and / or the like). For example, merchant system 304, which may include a physical POS device, may communicate with payment gateway system 302 through a public or private network to conduct card-present transactions. As another example, merchant system 304 that includes a server (e.g., a web server) may communicate with payment gateway system 302 through a public or private network, such as the Internet, to conduct card-not-present transactions.

[0138] For the purpose of illustration, processing a transaction (e.g., a payment transaction) may include generating a transaction message (e.g., authorization request and / or the like) based on an account identifier of a customer (e.g., accountholder associated with consumer device 310 and / or the like) and / or transaction data associated with the transaction. For example, merchant system 304 (e.g., a client device of merchant system 304, a POS device of merchant system 304, and / or the like) may initiate the transaction, e.g., by generating an authorization request (e.g., in response to receiving the account identifier from a payment device and / or a portable financial device of the customer and / or the like). Merchant system 304 may communicate the authorization request to payment gateway system 302 and / or acquirer system 308. In some non-limiting embodiments or aspects, payment gateway system 302 may communicate the authorization request to acquirer system 308 and / or transaction processing system 301. Additionally or alternatively, acquirer system 308 (and / or payment gateway system 302) may communicate the authorization request to transaction processing system 301. After receiving the 63T8613.DOCX Page 34 of 55Attorney Docket No.08223-2500316 (8143WO01) authorization request from merchant system 304 that identifies the account identifier of the customer (e.g., the accountholder associated with consumer device 310 and / or the account identifier), transaction processing system 301 may communicate the authorization request to issuer system 306 (e.g., the issuer system that issued the payment device and / or account identifier). Issuer system 306 may determine an authorization decision (e.g., approve, deny, and / or the like) based on the authorization request, and / or issuer system 306 may generate an authorization response based on the authorization decision and / or the authorization request. Issuer system 306 may communicate the authorization response to transaction processing system 301. Transaction processing system 301 may communicate the authorization response to acquirer system 308 and / or payment gateway system 302. In some non-limiting embodiments or aspects, acquirer system 308 may communicate the authorization response to payment gateway system 302 and / or merchant system 304. Additionally or alternatively, payment gateway system 302 (and / or acquirer system 308) may communicate the authorization response to merchant system 304.

[0139] In some non-limiting embodiments or aspects, transaction processing system 301 and / or issuer system 306 may include at least one machine learning model (e.g., at least one of a fraud detection model, a risk detection model, a transaction authorization model, a credit approval model, a product recommendation model, a classifier model, an anomaly detection model, an authentication model, any combination thereof, and / or the like). For example, the machine learning model(s) may be trained and / or refined (e.g., based on generated data records), as described herein. Transaction processing system 301 and / or issuer system 306 may perform at least one task (e.g., generate at least one prediction) based on the authorization request and the machine learning model(s). For example, performing the task(s) may include generating at least one prediction associated with fraud detection, risk detection, transaction authorization, credit approval, product recommendation, classification, anomaly detection, authentication, any combination thereof, and / or the like. In some non-limiting embodiments or aspects, transaction processing system 301 may communicate at least one message based on performing the task (e.g., generating the prediction) to issuer system 306 (e.g., along with the authorization request). In some non-limiting embodiments or aspects, issuer system 306 may determine the authorization decision (e.g., approve, deny, and / or the like) based on the authorization request and the performance of the task (e.g., generation of the prediction). 63T8613.DOCX Page 35 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0140] In some non-limiting embodiments, a machine learning model (e.g., first model 104) may be deployed in a production environment. For example, transaction processing system 301 and / or issuer system 306 may deploy a pretrained machine learning model (e.g., first model 104) in payment processing network 300 to generate outputs based on incoming authorization requests, as described herein. At least one anomaly (e.g., a shift between a distribution of the incoming authorization requests with respect to a distribution of the training data, a shift between a first distribution of the incoming authorization requests for a first time period with respect to a second distribution of the incoming authorization requests for a second time period, an increase in an error rate of the outputs generated by the machine learning model based on the incoming authorization requests, any combination thereof, and / or the like) may be detected (e.g., by transaction processing system 301 and / or issuer system 306) in the production environment (e.g., payment processing network 300). Machine learning model system 102 may receive the machine learning model (e.g., first model 104). For example, machine learning model system 102 may be in a development environment (e.g., of transaction processing system 301 and / or issuer system 306). Machine learning model system 102 may refine the machine learning model (e.g., first model 104), as described herein. The refined machine learning model (e.g., first model 104 and / or a refined version thereof, as described herein) may be deployed in the production environment (e.g., deployed by transaction processing system 301 and / or issuer system 306 in payment processing network 300).

[0141] For the purpose of illustration, clearing and / or settlement of a transaction may include generating a message (e.g., clearing message and / or the like) based on an account identifier of a customer (e.g., associated with consumer device 310 and / or the like) and / or transaction data associated with the transaction. For example, merchant system 304 may generate at least one clearing message (e.g., a plurality of clearing messages, a batch of clearing messages, and / or the like). Merchant system 304 may communicate the clearing message(s) to acquirer system 308 (and / or payment gateway system 302, which may communicate the clearing message(s) to acquirer system 308). Acquirer system 308 may communicate the clearing message(s) to transaction processing system 301. Transaction processing system 301 may communicate the clearing message(s) to issuer system 306. Issuer system 306 may generate at least one settlement message based on the clearing message(s). In some non-limiting embodiments or aspects, issuer system 306 may communicate 63T8613.DOCX Page 36 of 55Attorney Docket No.08223-2500316 (8143WO01) the settlement message(s) and / or funds to transaction processing system 301 (and / or a settlement bank system associated with transaction processing system 301), and transaction processing system 301 (and / or the settlement bank system) may communicate the settlement message(s) and / or funds to acquirer system 308. Additionally or alternatively, issuer system 306 may communicate the settlement message(s) and / or funds to acquirer system 308. In some non-limiting embodiments or aspects, acquirer system 308 may communicate settlement message(s) and / or funds to merchant system 304 (and / or an account associated with merchant system 304).

[0142] The systems and / or devices of FIG. 3 may communicate via one or more wired and / or wireless communication networks. For example, the communication network(s) may include a cellular network (e.g., a long-term evolution (LTE®) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, and / or the like), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and / or the like, and / or a combination of these or other types of networks.

[0143] The number and arrangement of systems, devices, and / or networks shown in FIG. 3 are provided as an example. There may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or differently arranged systems, devices, and / or networks than those shown in FIG.3. Furthermore, two or more systems or devices shown in FIG.3 may be implemented within a single system or device, or a single system or device shown in FIG. 3 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of payment processing network 300 may perform one or more functions described as being performed by another set of systems or another set of devices of payment processing network 300.

[0144] Referring now to FIG. 4, shown is a diagram of example components of device 400, according to some non-limiting embodiments or aspects. Device 400 may correspond to machine learning model system 102, database 108, and / or user device 63T8613.DOCX Page 37 of 55Attorney Docket No.08223-2500316 (8143WO01) 110 of FIG. 1 and / or transaction processing system 301, payment gateway system 302, merchant system 304, issuer system 306, acquirer system 308, and / or consumer device 310 of FIG.3, as an example. In some non-limiting embodiments, such systems or devices may include at least one device 400 and / or at least one component of device 400. The number and arrangement of components shown are provided as an example. In some non-limiting embodiments, device 400 may include additional components, fewer components, different components, or differently arranged components than those shown. Additionally or alternatively, a set of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another set of components of device 400.

[0145] As shown in FIG. 4, device 400 may include bus 402, processor 404, memory 406, storage component 408, input component 410, output component 412, and communication interface 414. Bus 402 may include a component that permits communication among the components of device 400. In some non-limiting embodiments, processor 404 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 404 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 406 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 404.

[0146] With continued reference to FIG. 4, storage component 408 may store information and / or software related to the operation and use of device 400. For example, storage component 408 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 410 may include a component that permits device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally, or alternatively, input component 410 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 412 may include a component that 63T8613.DOCX Page 38 of 55Attorney Docket No.08223-2500316 (8143WO01) provides output information from device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 414 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 400 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 414 may permit device 400 to receive information from another device and / or provide information to another device. For example, communication interface 414 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0147] Device 400 may perform one or more processes described herein. Device 400 may perform these processes based on processor 404 executing software instructions stored by a computer-readable medium, such as memory 406 and / or storage component 408. A computer-readable medium may include any non-transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 406 and / or storage component 408 from another computer-readable medium or from another device via communication interface 414. When executed, software instructions stored in memory 406 and / or storage component 408 may cause processor 404 to perform one or more processes described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0148] Referring now to FIGS. 5A-5D, shown are schematic diagrams of an example implementation 500 of systems and methods for enhancing machine learning model performance without altering the model or acquiring additional data, according to some non-limiting embodiments or aspects. The number and arrangement of 63T8613.DOCX Page 39 of 55Attorney Docket No.08223-2500316 (8143WO01) components shown in FIGS. 5A-5D are provided as an example. There may be additional components, fewer components, different components, and / or differently arranged components than those shown in FIGS. 5A-5D. Furthermore, two or more components shown in FIGS.5A-5D may be implemented within a single component, or a single component shown in FIGS. 5A-5D may be implemented as multiple, distributed components. Additionally or alternatively, a set of components (e.g., one or more components) of implementation 500 may perform one or more functions described as being performed by another set of components of implementation 500. In some non-limiting embodiments or aspects, implementation 500 may be performed (e.g., completely, partially, etc.) by machine learning model system 102 (e.g., one or more devices of machine learning model system 102). In some non-limiting embodiments or aspects, at least a portion of implementation 500 may be performed (e.g., completely, partially, etc.) by another device or a group of devices separate from or including machine learning model system 102, such as database 108 and / or user device 110.

[0149] As shown in FIG. 5A, in some non-limiting embodiments or aspects, implementation 500 may include machine learning (ML) model 502, refined ML model 510, data 520, first data subset 522, second data subset 524, synthetic data generator 530, evaluator model 540, and / or synthetic data 550. In some non-limiting embodiments or aspects, ML model 502 may be the same as or similar to first model 104. In some non-limiting embodiments or aspects, refined ML model 510 may initially be the same as or similar to first model 104 (e.g., initialized to be the same as first model 104 before being refined, as described herein). In some non-limiting embodiments or aspects, data 520, first data subset 522, second data subset 524, and / or synthetic data 550 may be stored in database 108 and / or machine learning model system 102. In some non-limiting embodiments or aspects, evaluator model 540 may be the same as or similar to second model 106. In some non-limiting embodiments or aspects, synthetic data generator may be the same as, similar to, and / or part of machine learning model system 102.

[0150] In some non-limiting embodiments or aspects, data 520 may include at least one data set and known labels for each sample of data in the data set. The data set may be input to ML model 502 to generate predicted labels (e.g., predicted outputs of ML model 502). Each predicted label may be associated with a right decision (e.g., correctly labeled by ML model 502 such that the predicted label matches the known 63T8613.DOCX Page 40 of 55Attorney Docket No.08223-2500316 (8143WO01) label) or a wrong decision (e.g., incorrectly labeled by ML model 502 such that the predicted label is different than the known label). Incorrectly labeled data from the data set may be identified based on comparing the predicted labels with known labels for the data set. For example, the predicted labels associated with the incorrectly labeled data may be different than the known labels. In some non-limiting embodiments or aspects, first data subset 522 may include the incorrectly labeled data (e.g., data samples for which the predicted label from ML model 502 is different than the known label). Additionally or alternatively, second data subset 524 may include the correctly labeled data (e.g., data samples for which the predicted label from ML model 502 matches the known label).

[0151] In some non-limiting embodiments or aspects, ML model 502 may include a classifier model. For example, the classifier model may include at least one of a neural network, a deep neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder (or portion thereof), a long short-term memory (LSTM), a bidirectional LSTM, an attention network, a transformer, a tree- based classifier, a random forest, a gradient boosting classifier, a support vector machine (SVM), a one-class support vector machine (OCSVM), a deep support vector data description (DeepSVDD), any combination thereof, and / or the like.

[0152] In some non-limiting embodiments or aspects, ML model 502 may be a pretrained model (e.g., a machine learning model that was previously trained). For example, if ML model 502 is well-designed and well-trained, it may have relatively good performance such that most data samples of data 520 may be correctly labeled (e.g., the predicted label from ML model 502 matches the known label, implying that ML model 502 made a right decision). In such an example, second data subset 524 associated with correctly labeled data may be larger (e.g., much larger) than first data subset 522 associated with incorrectly labeled data. However, as there still may be some incorrectly labeled data, refining ML model 502 (e.g., to provide refined ML model 510), as described herein, may improve performance even more.

[0153] In some non-limiting embodiments or aspects, the incorrectly labeled data (e.g., first data subset 522) may be sparse (e.g., relatively rare in data 520, such that first data subset 522 is much smaller than second data subset 524). Synthetic data generator 530 may help to address the sparsity of incorrectly labeled data by generating additional synthetic data, as described herein, for refining ML model 502 (e.g., to provide refined ML model 510). Additionally or alternatively, the incorrectly 63T8613.DOCX Page 41 of 55Attorney Docket No.08223-2500316 (8143WO01) labeled data (e.g., first data subset 522) may be noisily labeled. For example, while many of the incorrectly labeled data samples will have a predicted label from ML model 502 that is different than the known label because ML model 502 made the wrong decision, some of the incorrectly labeled data samples may be have a predicted label from ML model 502 that is different than the known label because the known label is wrong (e.g., ML model 502 actually made a right decision, but the known label is wrong). Evaluator model 540 may help to address the noisiness of the labels by scoring both the predicted label and the known label, as described herein.

[0154] With reference to FIG. 5B, and with continued reference to FIG. 5A, evaluator model 540 may be trained based on the incorrectly labeled data (e.g., data samples 522a of first data subset 522), the predicted labels 504 associated with the incorrectly labeled data, and the known labels 522b associated with the incorrectly labeled data to generate at least one score (e.g., a label score (slabel) 542 associated with the accuracy of each label 522b and / or a model score (smodel) 544 associated with the accuracy of each predicted label 504) for each data sample 522a of the incorrectly labeled data (e.g., first data subset 522). In some non-limiting embodiments or aspects, evaluator model 540 may be trained based on the incorrectly labeled data (e.g., data samples 522a of first data subset 522), the predicted labels 504 associated with the incorrectly labeled data, and the known labels 522b associated with the incorrectly labeled data to generate both a label score (slabel) 542 and a model score (smodel) 544.

[0155] In some non-limiting embodiments or aspects, evaluator model 540 may include a scoring model. In some non-limiting embodiments or aspects, evaluator model 540 may include at least one of a neural network, a deep neural network, a CNN, an RNN, an autoencoder (or portion thereof), an LSTM, a bidirectional LSTM, an attention network, a transformer, an SVM, an OCSVM, a DeepSVDD, any combination thereof, and / or the like. In some non-limiting embodiments or aspects, evaluator model 540 may include the same type of model as ML model 502. In some non-limiting embodiments or aspects, evaluator model 540 may include a different type of model than ML model 502.

[0156] In some non-limiting embodiments or aspects, training may include inputting each respective data sample 522a of the training data set (e.g., first data subset 522) to evaluator model 540 to generate a label score (slabel) 542 and a model score (smodel) 544 for reach respective data sample 522a. A loss may be determined based on each 63T8613.DOCX Page 42 of 55Attorney Docket No.08223-2500316 (8143WO01) respective data sample 522a and the respective predicted scores (e.g., slabel 542 smodel 544). For example, the losses may be calculated based on a loss function, a reward function, a divergence, an error, a mean error, a mean squared error (MSE), any combination thereof, and / or the like. For the purpose of illustration, an example loss function may be represented by the following equation:where ℒ is the loss and ^^^() is the maximum function, which will return the greater of smodel - slabel or 0 (e.g., will return the difference between the labels if the score for the model is greater than the score for the label, or otherwise return 0).

[0157] The parameters of evaluator model 540 may be updated based on the loss. For example, machine learning model system 102 may update (e.g., adjust) the parameters of evaluator model 540 based on back propagation (e.g., of the loss), gradient calculations (e.g., based on the loss), any combination thereof, and / or the like.

[0158] In some non-limiting embodiments or aspects, training may further include repeating the training process (e.g., repeating inputting each respective data sample 522a of first data subset 522, determining the loss, and updating the parameters) until a termination condition is satisfied. For example, the termination condition may include a threshold level of performance (e.g., accuracy, precision, recall and / or the like), and the repetition may continue until the threshold level of performance is satisfied. Additionally or alternatively, the termination condition may include a target number of repetitions (e.g., a target number of epochs and / or the like), and the repetition may continue until the target number of repetitions is completed. In some non-limiting embodiments or aspects, there may be a plurality of termination conditions, and the repetition may continue until at least one of the plurality of termination conditions is satisfied.

[0159] With reference to FIG. 5C, and with continued reference to FIG. 5A, synthetic data generator 530 may generate synthetic data 550 based on the data set (e.g., data 520). For example, statistics extractor 532 (e.g., of synthetic data generator 530) may determine a distribution associated with at least one feature (e.g., each feature) of the data set (e.g., data 520). Additionally or alternatively, data generator 534 (e.g., of synthetic data generator 530) may generate synthetic data 550 based on the data set (e.g., based on the distribution of at least one feature of data 520). 63T8613.DOCX Page 43 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0160] In some non-limiting embodiments or aspects, statistics extractor 532 may determine a distribution for each feature of data set (e.g., data 520). For example, data 520 may include at least one (e.g., a plurality of) numeric feature(s), at least one (e.g., a plurality of) nominal feature(s) (e.g., categorical feature and / or the like), any combination thereof, and / or the like. For each numerical feature, statistics extractor 532 may determine a mean value and a deviation value of the numeric feature in data 520, and the distribution for such a numerical feature may be associated with the mean and deviation values (e.g., a normal distribution ^ based on the mean and deviation values and / or the like). For each nominal feature, statistics extractor 532 may determine all of the unique values (e.g., categories and / or the like) of the nominal feature in data 520, determine a count of each unique value in data 520, and normalize the counts (e.g., calculate a proportion of data 520 associated with each unique value). The normalized values may be used as probabilities for a binomial or multinomial distribution. For the purpose of illustration, if 150 data samples 522a are associated with a first category of a selected feature and 450 data samples 522a are associated with a second category of the selected feature, then the normalized values for the first and second categories may be 0.25 and 0.75, respectively, and these normalized values may be used as the probabilities for a binomial distribution.

[0161] In some non-limiting embodiments or aspects, synthetic data 550 may include a plurality of synthetic data samples. In some non-limiting embodiments or aspects, data generator 534 may select a sample from the data set (e.g., data 520). For example, data generator 534 may select a sample from data 520 based on a uniform distribution of all data samples in data 520. In some non-limiting embodiments or aspects, data generator 534 may select a feature of the sample. For example, data generator 534 may select the feature based on a uniform distribution of all features in data 520. In some non-limiting embodiments or aspects, data generator 534 may sample a synthetic value for the feature based on a distribution associated with the feature. For example, if the feature is a numeric feature, data generator 534 may sample a numeric value from a normal distribution based on the mean and deviation values of that numeric feature. If the feature is a nominal feature, data generator 534 may sample a value (e.g., categorical value and / or the like) based on the binomial and / or multinomial distribution of that nominal feature. In some non-limiting embodiments or aspects, data generator 534 may replace a current value of the feature with the synthetic value for the feature in the selected sample to provide the 63T8613.DOCX Page 44 of 55Attorney Docket No.08223-2500316 (8143WO01) synthetic data sample. For example, a synthetic data sample may be a copy of the sample with the synthetic value substituted for the current value. In this way, synthetic data generator 530 may generate a large amount of unseen data (with similar distribution to existing data), which may be used to refine ML model 502 (e.g., to provide refined ML model 510), as described herein, which may make refined ML model 510 more robust (e.g., compared to ML model 502) and / or improve the performance of refined ML model 510 (e.g., compared to ML model 502).

[0162] With reference to FIG.5D, and with continued reference to FIG.5A, refined ML model 510 may be initialized. For example, refined ML model 510 may be initialized based on ML model 502. In some non-limiting embodiments or aspects, initializing refined ML model 510 may include copying the parameters ^^^^^^^^as the parameters^^ of refined ML model 510 (e.g., ^^ = ^^^^^^^^ ).

[0163] In some non-limiting embodiments or aspects, fixed ML model 502a may be the same as or similar to ML model 502. In some non-limiting embodiments or aspects, fixed ML model 502a may be initialized based on ML model 502. In some non-limiting embodiments or aspects, initializing fixed ML model 502a may include copying theparameters ^^^^^^^^ as the parameters ^! of fixed ML model 510 (e.g., ^! = ^^^^^^^^ ).

[0164] In some non-limiting embodiments or aspects, synthetic data 550 (e.g., synthetic data samples 552a thereof) may be inputted to fixed ML model 502a to generate synthetic predicted labels 504a (e.g., a respective synthetic predicted label 504a for each synthetic data sample 552a of synthetic data 550). Additionally or alternatively, synthetic data 550 (e.g., synthetic data samples 552a thereof) may be inputted to refined ML model 510 to generate refined synthetic predicted labels 514 (e.g., a respective refined synthetic predicted label 514 for each synthetic data sample 552a of synthetic data 550).

[0165] In some non-limiting embodiments or aspects, evaluator model 540 may generate at least one score based on at least one predicted label(s) (e.g., synthetic predicted label(s) 504a and / or refined synthetic predicted label(s) 514). For example, evaluator model 540 may generate first scores based on synthetic predicted labels 504a (e.g., generated by fixed ML model 502a). Additionally or alternatively, evaluator model 540 may generate second scores based on refined synthetic predicted label 514 (e.g., generated by refined ML model 510). 63T8613.DOCX Page 45 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0166] In some non-limiting embodiments or aspects, refined ML model 510 may be trained based on the score(s). For example, a loss may be determined based on the first score based on synthetic predicted label 504a (e.g., generated by fixed ML model 502a) and the second score based on refined synthetic predicted label 514 (e.g., generated by refined ML model 510) for each synthetic data sample 552a. For example, the loss may be calculated based on a loss function, a reward function, a divergence, an error, a mean error, an MSE, any combination thereof, and / or the like. For the purpose of illustration, an example loss function may be represented by the following equation:where ℒ is the loss, ^ is a synthetic data sample 552a of a set 1 of synthetic data 550, #() is the evaluator model 540, %&() is the fixed ML model 502a with parameters ^&, %(() is the refined ML model 510 with parameters ^(, . is a hyperparameter coefficient (e.g., a positive number and / or the like), ^^^() is the maximum function, and KL() is the Kullback–Leibler (KL) divergence.

[0167] The parameters ^(of refined ML model 510 may be updated based on the loss. For example, machine learning model system 102 may update (e.g., adjust) the parameters of refined ML model 510 based on back propagation (e.g., of the loss), gradient calculations (e.g., based on the loss), any combination thereof, and / or the like. In some non-limiting embodiments or aspects, the parameters ^&of fixed ML model 502a may be fixed (e.g., may remain constant) while the parameters ^(of refined ML model 510 are updated.

[0168] In some non-limiting embodiments or aspects, updating refined model based on a set 1 of synthetic data 550, as described above, may be an inner round of training. In some non-limiting embodiments or aspects, after completing an inner round of training, the parameters ^(of refined ML model 510 may be copied as the parameters ^&of fixed ML model 502a, which may be an outer round of training. After each repetition of the outer round of training is completed, another inner round of training may be repeated. In some non-limiting embodiments or aspects, during each inner round, the parameters ^&of fixed ML model 502a may be fixed (e.g., may remain constant) while the parameters ^(of refined ML model 510 are updated. 63T8613.DOCX Page 46 of 55Attorney Docket No.08223-2500316 (8143WO01)

[0169] In some non-limiting embodiments or aspects, the training process may be repeated (e.g., inner rounds and outer rounds of training may be repeated) until a termination condition is satisfied. For example, the termination condition may include a threshold level of performance (e.g., accuracy, precision, recall and / or the like), and the repetition may continue until the threshold level of performance is satisfied. Additionally or alternatively, the termination condition may include a target number of repetitions (e.g., a target number of inner rounds, a target number of outer rounds, any combination thereof, and / or the like), and the repetition may continue until the target number of repetitions is completed. In some non-limiting embodiments or aspects, there may be a plurality of termination conditions, and the repetition may continue until at least one of the plurality of termination conditions is satisfied.

[0170] In some non-limiting embodiments or aspects, the training process may result in reinforcement learning (RL), by which the refined ML model 510 is improved with respect to the fixed ML model 502a through each repetition of the training process. By using the synthetic data 550 as the training data and generating new synthetic data 550 for each (inner) round of the training process, the feature space may be fully explored and / or unknown areas of the feature space may be exploited (e.g., for training of refined ML model 510). In this way, refined ML model 510 may have improved performance and robustness compared to ML model 502. Additionally, by generating synthetic data 550, these improvements may be achieved without the need to obtain more real data. Also, refined ML model 510 does not require changing the design of ML model 502 (e.g., the models have the same design and the parameters from ML model 502 are initially copied to initialized refined ML model 510). The disclosed techniques are also flexible and may be applied to various types of machine learning models, as described herein. Moreover, the evaluator model 540 may be the same type of model or a different type of model with respect to ML model 502, and, therefore, the evaluator model 540 may be selected and / or designed to increase the improvement in performance of refined ML model 510 with respect to ML model 502.

[0171] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present disclosure contemplates that, to the extent possible, one or more features of 63T8613.DOCX Page 47 of 55Attorney Docket No.08223-2500316 (8143WO01) any embodiment or aspect can be combined with one or more features of any other embodiment or aspect. 63T8613.DOCX Page 48 of 55

Claims

Attorney Docket No.08223-2500316 (8143WO01) WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: inputting, with at least one processor, a data set to a first machine learning model to generate predicted labels; identifying, with at least one processor, incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; training, with at least one processor, a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generating, with at least one processor, synthetic data based on the data set; and refining, with at least one processor, the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

2. The method of claim 1, wherein the first machine learning model comprises a classifier model.

3. The method of claim 1, wherein the second machine learning model comprises at least one of an evaluator model, a scoring model, or any combination thereof.

4. The method of claim 1, wherein the score associated with accuracy of each predicted label associated with the incorrectly labeled data comprises a first score, and wherein training comprises training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled 63T8613.DOCX Page 49 of 55Attorney Docket No.08223-2500316 (8143WO01) data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

5. The method of claim 1, further comprising: inputting, with at least one processor, the synthetic data to the first machine learning model to generate the synthetic predicted labels.

6. The method of claim 1, wherein the refining comprises: initializing a refined model based on the first machine learning model; and training the refined machine learning model.

7. The method of claim 6, wherein the scores associated with the synthetic predicted labels comprise first scores, wherein the synthetic predicted labels associated with the synthetic data generated by the first machine learning model comprise first synthetic predicted labels, and wherein training the refined machine learning model comprises: inputting, with at least one processor, the synthetic data to the first machine learning model to generate the first synthetic predicted labels; inputting, with at least one processor, the synthetic data to the refined machine learning model to generate second synthetic predicted labels; inputting, with at least one processor, the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels; and updating, with at least one processor, parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

8. The method of claim 1, wherein the synthetic data comprises a plurality of synthetic data samples, and wherein generating the synthetic data 63T8613.DOCX Page 50 of 55Attorney Docket No.08223-2500316 (8143WO01) comprises generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set; selecting a feature of the sample; sampling a synthetic value for the feature based on a distribution associated with the feature; and replacing a current value of the feature with the synthetic value for the feature in the sample, wherein the synthetic data sample comprises the sample with the synthetic value.

9. The method of claim 8, wherein at least one feature of the sample comprises a numeric feature, and wherein the distribution associated with each numeric feature is associated with a mean value and a deviation value.

10. The method of claim 8, wherein at least one feature of the sample comprises a nominal feature, and wherein the distribution associated with each nominal feature is associated with at least one of a binomial distribution, a multinomial distribution, or any combination thereof.

11. A system comprising: at least one processor configured to: input a data set to a first machine learning model to generate predicted labels; identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; train a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generate synthetic data based on the data set; and 63T8613.DOCX Page 51 of 55Attorney Docket No.08223-2500316 (8143WO01) refine the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model.

12. The system of claim 11, wherein the first machine learning model comprises a classifier model.

13. The system of claim 11, wherein the second machine learning model comprises at least one of an evaluator model, a scoring model, or any combination thereof.

14. The system of claim 11, wherein the score associated with accuracy of each predicted label associated with the incorrectly labeled data comprises a first score, and wherein training comprises training the second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate the first score associated with accuracy of each predicted label associated with the incorrectly labeled data and a second score associated with accuracy of each known label associated with the incorrectly labeled data.

15. The system of claim 11, wherein the at least one processor is further configured to: input the synthetic data to the first machine learning model to generate the synthetic predicted labels.

16. The system of claim 11, wherein the refining comprises: initializing a refined model based on the first machine learning model; and training the refined machine learning model.

17. The system of claim 16, wherein the scores associated with the synthetic predicted labels comprise first scores, 63T8613.DOCX Page 52 of 55Attorney Docket No.08223-2500316 (8143WO01) wherein the synthetic predicted labels associated with the synthetic data generated by the first machine learning model comprise first synthetic predicted labels, and wherein training the refined machine learning model comprises: inputting the synthetic data to the first machine learning model to generate the first synthetic predicted labels; inputting the synthetic data to the refined machine learning model to generate second synthetic predicted labels; inputting the synthetic data, the first synthetic predicted labels, and the second synthetic predicted labels to the second machine learning model to generate the first scores associated with the first synthetic predicted labels and second scores associated with the second synthetic predicted labels; and updating parameters of the refined machine learning model based on a loss based on the first scores and the second scores.

18. The system of claim 11, wherein the synthetic data comprises a plurality of synthetic data samples, and wherein generating the synthetic data comprises generating each synthetic data sample of the plurality of synthetic data samples by: selecting a sample from the data set; selecting a feature of the sample; sampling a synthetic value for the feature based on a distribution associated with the feature; and replacing a current value of the feature with the synthetic value for the feature in the sample, wherein the synthetic data sample comprises the sample with the synthetic value.

19. The system of claim 18, wherein at least one feature of the sample comprises a numeric feature, and wherein the distribution associated with each numeric feature is associated with a mean value and a deviation value. 63T8613.DOCX Page 53 of 55Attorney Docket No.08223-2500316 (8143WO01) 20. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: input a data set to a first machine learning model to generate predicted labels; identify incorrectly labeled data from the data set based on comparing the predicted labels with known labels for the data set, wherein the predicted labels associated with the incorrectly labeled data are different than the known labels; train a second machine learning model based on the incorrectly labeled data, the predicted labels associated with the incorrectly labeled data, and the known labels associated with the incorrectly labeled data to generate a score associated with accuracy of each predicted label associated with the incorrectly labeled data; generate synthetic data based on the data set; and refine the first machine learning model based on the synthetic data, synthetic predicted labels associated with the synthetic data generated by the first machine learning model, and scores associated with the synthetic predicted labels generated by the second machine learning model. 63T8613.DOCX Page 54 of 55

Images