Methods, systems, and computer program products for improving machine learning models

Abstract

Methods, systems, and computer program products are provided for improving machine learning models, the method comprising: receiving a dataset comprising data records; inputting the dataset to a first pre-trained machine learning model to generate first embeddings; inputting the first embeddings to a second machine learning model to generate second embeddings in a user-specific embedding space; inputting the plurality of second embeddings to a third machine learning model to extract feature data associated with features; inputting output from a machine learning system and the feature data to a fourth machine learning model to generate a relevance score for each entity; determining a subset of entities based on the relevance score; transmitting a request for feedback to a user; receiving feedback data from the user; and training at least one of the models based on the feedback data.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application 63 / 437,013, filed January 4, 2023, the disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND 1. TECHNICAL FIELD

[0004] The present disclosure relates generally to analyzing and / or improving machine learning models, and in some non-limiting embodiments or aspects, to methods, systems, and computer program products for analyzing and / or improving machine learning models.

[0005] 2. TECHNICAL CONSIDERATIONS

[0006] Machine learning models are powerful tools that are capable of identifying patterns and / or generating decisions based on a dataset. Machine learning models can be trained by inputting training data and updating the parameters of the model based on the output generated by the model in response to the training data. However, these models can be error-prone and can be difficult to maintain the performance of the model over time. For example, the performance of the model can decrease over time due to lack of supervision. SUMMARY

[0007] Accordingly, methods, systems, and computer program products are provided for analyzing and / or improving machine learning models (e.g., overcoming some or all of the above-identified shortcomings).

[0008] According to non-limiting embodiments or aspects, a computer-implemented method is provided. The method can include receiving, with at least one processor, a data set comprising a plurality of data records. Each data record of the plurality of data records can be associated with an entity of a plurality of entities. The method can include inputting, with at least one processor, the data set to a first pre-trained machine learning model to generate a plurality of first embedding vectors. Each respective first embedding vector of the plurality of first embedding vectors can be associated with a respective entity of the plurality of entities. The method can include inputting, with at least one processor, the plurality of first embedding vectors to a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user. The method can include inputting, with at least one processor, the plurality of second embedding vectors to a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors. The method can include inputting, with at least one processor, the output from the machine learning system and the feature data to a fourth machine learning model to generate a relevance score for each entity of the plurality of entities. The method can include determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities. The method can include transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities. The method can include receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device. The method can include training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0009] In some non-limiting embodiments or aspects, training at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof can include modifying at least one parameter of at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0010] In some non-limiting embodiments or aspects, the feedback data can include a binary classification for each entity of the subset of entities.

[0011] In some non-limiting embodiments or aspects, the data set can include multivariate time series data having a plurality of data records, each respective data record of the plurality of data records including time-ordered data elements for a plurality of variables.

[0012] In some non-limiting embodiments or aspects, the dataset can include transaction data associated with a plurality of transactions, each data record associated with at least one transaction of the plurality of transactions.

[0013] In some non-limiting embodiments or aspects, the correlation score can be associated with at least one of: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0014] In some non-limiting embodiments or aspects, training at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof can include determining, with the at least one processor, a loss based on the feedback data and the loss function, and training, with the at least one processor, at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the loss.

[0015] In some non-limiting embodiments or aspects, the loss function can include at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their correlation scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their correlation scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with an incorrect relative ranking between an entity in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

[0016] In some non-limiting embodiments or aspects, the loss function can further include a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component. The subset of entities can include a predetermined number of entities having one of a highest correlation score or a lowest correlation score.

[0017] According to non-limiting embodiments or aspects, a system is provided that includes at least one processor configured to: receive a data set including a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; input the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; input the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; input the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; input the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; determine a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmit a feedback request to a user device of the user based on the subset of entities; receive feedback data for each entity of the subset of entities from the user device; and train at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0018] In some non-limiting embodiments or aspects, training at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, can include modifying at least one parameter of at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, based on the feedback data.

[0019] In some non-limiting embodiments or aspects, the feedback data can include a binary classification for each entity of the subset of entities.

[0020] In some non-limiting embodiments or aspects, the data set can include multivariate time series data having a plurality of data records, each respective data record of the plurality of data records including time-ordered data elements for a plurality of variables.

[0021] In some non-limiting embodiments or aspects, the data set can include transaction data associated with a plurality of transactions, each data record being associated with at least one transaction of the plurality of transactions.

[0022] In some non-limiting embodiments or aspects, the relevance score can be associated with at least one of the following: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0023] In some non-limiting embodiments or aspects, training at least one of the second machine-learned model, the third machine-learned model, the fourth machine-learned model, or any combination thereof can include the at least one processor being configured to: determine a loss based on the feedback data and a loss function; and train at least one of the second machine-learned model, the third machine-learned model, the fourth machine-learned model, or any combination thereof based on the loss.

[0024] In some non-limiting embodiments or aspects, the loss function can include at least one of: a first component to widen a gap between pairs of entities in the entity subset having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the entity subset having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the entity subset having feedback data associated with incorrect relative rankings between the entities in the entity subset and other entities in the plurality of entities not included in the entity subset; or any combination thereof.

[0025] In some non-limiting embodiments or aspects, the loss function can further include a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0026] In some non-limiting embodiments or aspects, the entity subset can include a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

[0027] According to non-limiting embodiments or aspects, there is provided a computer program product comprising at least one non-transitory computer-readable medium having program instructions, the program instructions, when executed by at least one processor, cause the at least one processor to: receive a data set comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; input the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; input the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; input the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; input the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; determine a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmit a feedback request to a user device of the user based on the subset of entities; receive feedback data for each entity of the subset of entities from the user device; and train at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0028] In some non-limiting embodiments or aspects, training at least one of the following: the second machine learning model, the third machine learning model, or the fourth machine learning model can comprise modifying at least one parameter of at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0029] In some non-limiting embodiments or aspects, the feedback data can comprise a binary classification for each entity of the subset of entities.

[0030] In some non-limiting embodiments or aspects, the data set can comprise multivariate time series data having a plurality of data records, each respective data record of the plurality of data records comprising time-ordered data elements for a plurality of variables.

[0031] In some non-limiting embodiments or aspects, the data set can comprise transaction data associated with a plurality of transactions, each data record being associated with at least one transaction of the plurality of transactions.

[0032] In some non-limiting embodiments or aspects, the relevance score can be associated with at least one of: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0033] In some non-limiting embodiments or aspects, training at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof can include the program instructions causing the at least one processor to: determine a loss based on the feedback data and a loss function; and train at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the loss.

[0034] In some non-limiting embodiments or aspects, the loss function can include at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with an incorrect relative ranking between an entity in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

[0035] In some non-limiting embodiments or aspects, the loss function can further include a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0036] In some non-limiting embodiments or aspects, the subset of entities can include a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

[0037] According to non-limiting embodiments or aspects, a computer-implemented method is provided that includes: receiving, with at least one processor, a data set comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; inputting, with at least one processor, the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; inputting, with at least one processor, the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; inputting, with at least one processor, the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; inputting, with at least one processor, at least one of the following into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities: an output from the machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, the feature data, or any combination thereof; determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities; receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device; and training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0038] In some non-limiting embodiments or aspects, at least two of the output from the machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, and the feature data can be inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0039] In some non-limiting embodiments or aspects, the output from the machine learning system can be inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0040] In some non-limiting embodiments or aspects, the plurality of first embedding vectors can be inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0041] In some non-limiting embodiments or aspects, the plurality of second embedding vectors can be inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0042] In some non-limiting embodiments or aspects, the feature data can be input to a fourth machine learning model to generate a relevance score for each entity of the plurality of entities.

[0043] Other non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0044] Clause 1 : A computer-implemented method comprising: receiving, with at least one processor, a dataset comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; inputting, with at least one processor, the dataset to a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; inputting, with at least one processor, the plurality of first embedding vectors to a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; inputting, with at least one processor, the plurality of second embedding vectors to a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; inputting, with at least one processor, the output from the machine learning system and the feature data to a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities; receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device; and training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0045] Clause 2: The method of clause 1, wherein training at least one of the following: the second machine learning model, the third machine learning model, or the fourth machine learning model comprises modifying at least one parameter of at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the feedback data.

[0046] Clause 3: The method of clause 1 or 2, wherein the feedback data comprises a binary classification for each entity of the subset of entities.

[0047] Clause 4: The method of any of clauses 1-3, wherein the dataset comprises multivariate time series data having a plurality of data records, each respective data record of the plurality of data records comprising time-ordered data elements for a plurality of variables.

[0048] Clause 5: The method of any of clauses 1-4, wherein the dataset comprises transaction data associated with a plurality of transactions, each data record associated with at least one transaction of the plurality of transactions.

[0049] Clause 6: The method of any of clauses 1-5, wherein the correlation score is associated with at least one of: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0050] Clause 7: The method of any of clauses 1-6, wherein training at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, comprises: determining, with the at least one processor, a loss based on the feedback data and a loss function, and training, with the at least one processor, at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, based on the loss.

[0051] Clause 8: The method of any of clauses 1-7, wherein the loss function comprises at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their correlation scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their correlation scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with an incorrect relative ranking between an entity in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

[0052] Clause 9: The method of any of clauses 1-8, wherein the loss function further comprises a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0053] Clause 10: The method of any of clauses 1-9, wherein the subset of entities comprises a predetermined number of entities having one of a highest correlation score or a lowest correlation score.

[0054] Clause 11: A system comprising at least one processor configured to: receive a data set comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; input the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; input the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; input the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; input the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; determine a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmit a feedback request to a user device of the user based on the subset of entities; receive feedback data for each entity of the subset of entities from the user device; and train at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0055] Clause 12: The system of clause 11, wherein training at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, comprises modifying at least one parameter of at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, based on the feedback data.

[0056] Clause 13: The system of clause 11 or 12, wherein the feedback data comprises a binary classification for each entity of the subset of entities.

[0057] Clause 14: The system of any of clauses 11-13, wherein the data set comprises multivariate time series data having a plurality of data records, each respective data record of the plurality of data records comprising time-ordered data elements for a plurality of variables.

[0058] Clause 15: The system of any of clauses 11-14, wherein the data set comprises transaction data associated with a plurality of transactions, each data record being associated with at least one transaction of the plurality of transactions.

[0059] Clause 16: The system of any of clauses 11-15, wherein the relevance score is associated with at least one of the following: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0060] Clause 17: The system of any of clauses 11-16, wherein training at least one of: the second machine-learned model, the third machine-learned model, the fourth machine-learned model, or any combination thereof, comprises: determining, by the at least one processor, a loss based on the feedback data and a loss function; and training, by the at least one processor, at least one of: the second machine-learned model, the third machine-learned model, the fourth machine-learned model, or any combination thereof, based on the loss.

[0061] Clause 18: The system of any of clauses 11-17, wherein the loss function comprises at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with incorrect relative rankings between entities in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

[0062] Clause 19: The system of any of clauses 11-18, wherein the loss function further comprises a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0063] Clause 20: The system of any of clauses 11-19, wherein the subset of entities comprises a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

[0064] Clause 21 : A computer program product comprising at least one non-transitory computer-readable medium having program instructions that, when executed by at least one processor, cause the at least one processor to: receive a data set comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; input the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; input the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; input the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; input the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; determine a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmit a feedback request to a user device of the user based on the subset of entities; receive feedback data for each entity of the subset of entities from the user device; and train at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0065] Clause 22: The computer program product of clause 21, wherein training at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, comprises modifying at least one parameter of at least one of the following: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, based on the feedback data.

[0066] Clause 23: The computer program product of clause 21 or 22, wherein the feedback data comprises a binary classification for each entity of the subset of entities.

[0067] Clause 24: The computer program product of any one of clauses 21-23, wherein the data set comprises multivariate time series data having a plurality of data records, each respective data record of the plurality of data records comprising time-ordered data elements for a plurality of variables.

[0068] Clause 25: The computer program product of any one of clauses 21-24, wherein the data set comprises transaction data associated with a plurality of transactions, each data record being associated with at least one transaction of the plurality of transactions.

[0069] Clause 26: The computer program product of any one of clauses 21-25, wherein the relevance score is associated with at least one of: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof.

[0070] Clause 27: The computer program product of any one of clauses 21-26, wherein training at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, comprises: the program instructions causing the at least one processor to: determine a loss based on the feedback data and a loss function; and train at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, based on the loss.

[0071] Clause 28: The computer program product of any one of clauses 21-27, wherein the loss function comprises at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with an incorrect relative ranking between an entity in the subset of entities and other entities in the plurality of entities that are not included in the subset of entities; or any combination thereof.

[0072] Clause 29: The computer program product of any one of clauses 21-28, wherein the loss function further comprises a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0073] Clause 30: The computer program product of any one of clauses 21-29, wherein the subset of entities comprises a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

[0074] Clause 31: A computer-implemented method comprising: receiving, with at least one processor, a dataset comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities; inputting, with at least one processor, the dataset into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity of the plurality of entities; inputting, with at least one processor, the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; inputting, with at least one processor, the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; inputting, with at least one processor, at least one of the following into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities: an output from the machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, the feature data, or any combination thereof; determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities; receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device; and training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

[0075] Clause 32: The computer-implemented method of clause 31, wherein at least two of the output from the machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, and the feature data are inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0076] Clause 33: The computer-implemented method of clause 31 or 32, wherein the output from the machine learning system is inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0077] Clause 34: The computer-implemented method of any of clauses 31-33, wherein the plurality of first embedding vectors are inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0078] Clause 35: The computer-implemented method of any of clauses 31-34, wherein the plurality of second embedding vectors are inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

[0079] Clause 36: The computer-implemented method of any of clauses 31-35, wherein the feature data is input to a fourth machine learning model to generate a relevance score for each of the plurality of entities.

[0080] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate similar, but not necessarily identical components. It is to be expressly understood, however, that the drawings are for the purpose of illustration only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0081] Additional advantages and details of implementation will now be explained with reference to the non-limiting example embodiments illustrated in the drawings, in which:

[0082] Figure 1 is a schematic diagram of an example system for analyzing and / or improving a machine learning model in accordance with some non-limiting embodiments or aspects;

[0083] Figure 2 is a schematic diagram of an example system for analyzing and / or improving a machine learning model in accordance with some non-limiting embodiments or aspects;

[0084] Figure 3 is a schematic diagram of a system for generating entity embeddings in accordance with some non-limiting embodiments or aspects;

[0085] Figure 4 is a schematic diagram of a system for generating user embeddings in accordance with some non-limiting embodiments or aspects;

[0086] Figure 5 is a schematic diagram of a system for generating user embeddings in accordance with some non-limiting embodiments or aspects;

[0087] Figure 6 is a schematic diagram of a system for submitting a feedback request in accordance with some non-limiting embodiments or aspects;

[0088] Figure 7 is a schematic diagram of an example system for analyzing and / or improving a machine learning model for a payment network in accordance with some non-limiting embodiments or aspects;

[0089] Figure 8 is a schematic diagram of a payment network in accordance with some non-limiting embodiments or aspects; Figure 7

[0090] Figure 9 ​is a flowchart of a method for analyzing and / or improving a machine learning model according to some non-limiting embodiments or aspects;

[0091] Figure 10 is a schematic diagram of an example environment in which the methods, systems, and / or computer program products described herein can be implemented according to some non-limiting embodiments or aspects;

[0092] Figure 11 is a schematic diagram of example components of one or more devices of Figures 1-8 and / or Figure 10

[0093] Figures 12A-12B show plots of Q-time accuracy and Q-time average accuracy as a function of steps for Q = 10 and Q = 20, respectively;

[0094] Figure 13 show plots of Q-time accuracy as a function of steps for Q = 10 and Q = 20 for various user embedder layer types; and

[0095] Figure 14 show loss function term sensitivity analysis. DETAILED DESCRIPTION

[0096] For the following description, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the orientation of the embodiments in the drawings. However, it is to be understood that the embodiments can be oriented in various alternative manners and that the specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0097] It is to be understood that the present disclosure can be taken in various alternative sequences and / or steps, except where explicitly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0098] Some non-limiting embodiments or aspects are described herein in connection with a threshold value. As used herein, satisfying a threshold value can refer to a value that is greater than the threshold value, more than the threshold value, higher than the threshold value, greater than or equal to the threshold value, less than the threshold value, less than the threshold value, lower than the threshold value, less than or equal to the threshold value, equal to the threshold value, or the like.

[0099] ​No aspect, component, element, structure, act, step, function, instruction, etc. used in the description or claims should be construed as critical or essential unless explicitly stated as such. Also, as used herein, the article “a” is intended to include one or more items, and can be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.), and can be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the term “has” and its variants are intended to identify an existence, rather than an explicit description of an item. Additionally, unless otherwise explicitly stated, the phrase “based on” is construed as meaning “based at least in part on.” Additionally, references to an action being “based on” a condition can mean that the action is “in response to” the condition. For example, in some non-limiting embodiments or aspects, the phrases “based on” and “in response to” can refer to a condition that automatically triggers an action (e.g., a particular operation of an electronic device such as a computing device, processor, etc.).

[0100] As used herein, the term “acquirer institution” can refer to an entity licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using payment devices associated with the transaction service provider. Transactions that an acquirer institution can initiate can include payment transactions (e.g., purchases, original credit transaction (OCT), account funding transaction (AFT), etc.). In some non-limiting embodiments or aspects, an acquirer institution can be a financial institution, such as a bank. As used herein, the term “acquirer system” can refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0101] As used herein, the term “account identifier” can include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term “token” can refer to an identifier used as a substitute or replacement identifier for an original account identifier such as a PAN. An account identifier can be any combination of alphanumeric or character and / or symbol. A token can be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) such that the token can be used to conduct transactions without the need to directly use the original account identifier. In some examples, an original account identifier such as a PAN can be associated with multiple tokens for different individuals or purposes.

[0102] As used herein, the terms “client” and “client device” can refer to one or more client-side devices or systems (e.g., at a remote of a transaction service provider) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a “client device” can refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and the like. In some non-limiting embodiments or aspects, a client device can be an electronic device configured to communicate with one or more networks and initiate or facilitate a transaction. For example, a client device can include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, and the like. Further, “client” can also refer to an entity (e.g., a merchant, an acquirer, etc.) that owns, utilizes, and / or operates a client device to initiate a transaction (e.g., to initiate a transaction with a transaction service provider).

[0103] As used herein, the term “communication” can refer to receipt, receipt of, transmission, transfer, provision, and the like of data (e.g., information, signals, messages, instructions, commands, etc.). That one unit (e.g., a device, a system, a component of a device or system, combinations thereof, etc.) is in communication with another unit means that the one unit is capable of directly or indirectly receiving a signal from and / or transmitting a signal to the other unit. This can refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, etc.) that is wired and / or wireless in nature. Additionally, two units can be in communication with each other even though the information transmitted can be modified, processed, relayed and / or routed through another unit. For example, a first unit can be in communication with a second unit even if the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit can be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message can refer to a network packet (e.g., a data packet, etc.) that includes data. It will be appreciated that there can be many other arrangements.

[0104] As used herein, the term“computing device” can refer to one or more electronic devices configured to process data. In some examples, a computing device can include the necessary components to receive, process, and output data, such as a processor, display, memory, input devices, network interfaces, etc. A computing device can be a mobile device. As an example, a mobile device can include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device can also be a desktop computer or other form of non-mobile computer.

[0105] As used herein, the terms“e-wallet” and“e-wallet application” refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an e-wallet can include a mobile device executing an e-wallet application, and can also include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An“e-wallet provider” can include an entity that provides and / or maintains an e-wallet for customers, such as Google Pay®, Android Pay®, Apple Pay®, Samsung Pay®, and / or other similar electronic payment systems. In some non-limiting examples, an issuer bank can be an e-wallet provider.

[0106] As used herein, the term“issuer institution” can refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions, such as payment transactions, e.g., initiating credit and / or debit payments. For example, an issuer institution can provide a customer with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with the customer. The account identifier can be embodied on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or can be electronic and used for electronic payments. The term“issuer system” refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system can include one or more authorization servers for authorizing transactions.

[0107] As used herein, the term“merchant” can refer to a person or entity that provides goods and / or services or access to goods and / or services to a customer based on a transaction, such as a payment transaction. The term“merchant” or“merchant system” can also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer executing one or more software applications.

[0108] As used herein, the term “payment device” can refer to a payment card (e.g., a credit card or a debit card), a gift card, a smart card, a smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or fob, an RFID transponder, a retailer discount or loyalty card, a cellular telephone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, or the like. In some non-limiting embodiments or aspects, a payment device can include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder name, and / or the like).

[0109] As used herein, the term “payment gateway” can refer to an entity and / or a payment processing system operated by or on behalf of such an entity (e.g., a merchant service provider, a payment service provider, a payment facilitator, a payment facilitator under contract with an acquirer, a payment aggregator, and / or the like) that provides payment services (e.g., transaction service provider payment services, payment processing services, and / or the like) to one or more merchants. The payment services can be associated with the use of portable financial devices managed by a transaction service provider. As used herein, the term “payment gateway system” can refer to one or more computer systems, computer devices, servers, groups of servers, and / or the like operated by or on behalf of a payment gateway.

[0110] As used herein, a “point of sale (POS) device” can refer to one or more devices that can be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device can include one or more client devices. Additionally or alternatively, a POS device can include a peripheral device, a card reader, a scanning device (e.g., a code scanner), a Bluetooth® communication receiver, a near-field communication (NFC) receiver, a radio-frequency identification (RFID) receiver, and / or other contactless transceivers or receivers, a contact-based receiver, a payment terminal, and / or the like. As used herein, a “point of sale (POS) system” can refer to one or more client devices and / or peripheral devices used by a merchant to conduct a transaction. For example, a POS system can include one or more POS devices, and / or other similar devices that can be used to conduct a payment transaction. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) can include one or more server computers configured to process online payment transactions through a web page, a mobile application, and / or the like.

[0111] As used herein, the term“server” can refer to or include one or more computing devices operated by or facilitating the communication and processing of a plurality of parties in a network environment, such as the Internet, although it will be appreciated that communication can be facilitated through one or more public or private network environments, and that various other arrangements are possible. Further, a plurality of computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment can constitute a“system.”

[0112] As used herein, the term“system” can refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components thereof, etc.). As used herein, references to“devices,”“servers,”“processors,” etc. can refer to devices, servers, or processors recited as performing previous steps or functions, different servers or processors, and / or combinations of servers and / or processors. For example, as used in the specification and claims, a first server or a first processor recited as performing a first step or a first function can refer to the same or different server or the same or different processor recited as performing a second step or a second function.

[0113] As used herein, the term“transaction service provider” can refer to an entity that receives transaction authorization requests from merchants or other entities and in some cases provides payment guarantees through an agreement between the transaction service provider and an issuer institution. For example, a transaction service provider can include a payment network, such as Visa®, or any other entity that processes transactions. The term“transaction processing system” can refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server that executes one or more software applications. A transaction processing server can include one or more processors and, in some non-limiting embodiments or aspects, can be operated by or on behalf of a transaction service provider.

[0114] Non-limiting embodiments or aspects of the disclosed subject matter are directed to systems, methods, and computer program products for analyzing and / or improving machine learning models. For example, non-limiting embodiments or aspects include a machine learning system that includes multiple machine learning models in separate layers that perform different tasks, which are combined such that a subsequent layer receives and relies on the output of a preceding layer, which improves the decisions generated by the machine learning models. The system can employ multiple machine learning models. A dataset can be input into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors being associated with a respective entity. The plurality of first embedding vectors can be input into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user. The plurality of second embedding vectors can be input into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors. The plurality of second embedding vectors and the feature data can be input into a fourth machine learning model to generate a relevance score for each entity.

[0115] Non-limiting embodiments or aspects of the disclosed subject matter include a feedback loop that receives user feedback to confirm or correct decisions of the model associated with potential anomalous outputs. Feedback data received from the feedback loop is used to train the machine learning models and layers to further improve the accuracy of the model. A loss function split into multiple components is employed by the system to train and improve the model, each component targeting a different aspect of the correction of the model.

[0116] The feedback data can improve the model output for entities such that the model makes more accurate predictions for events associated with the entities. The feedback data can improve the model output for users such that the model makes more accurate predictions for which user is most relevant to the entity and / or events of the entity, resulting in more accurate feedback from the most relevant user, further improving predictions for events associated with the entity. Thus, at least in part due to the unique and unconventional arrangement of the multiple machine learning models and layers and their use of feedback data from the feedback loop and the multi-component loss function, non-limiting embodiments or aspects of the present disclosure improve the accuracy of the machine learning models and the decisions produced thereby.

[0117] For purposes of illustration, in the following description, while the presently disclosed subject matter is described in terms of systems, methods, and computer program products for analyzing and / or improving machine learning models, those skilled in the art will recognize that the disclosed subject matter is not limited to the illustrative embodiments.

[0118] Reference Figure 1FIG. 1 shows an example system 100 for analyzing and / or improving a machine learning model, in accordance with some non-limiting embodiments or aspects. The system 100 can include a database 102, a machine learning system 104, and a querying device 106.

[0119] The database 102 can include at least one computing device, as described herein. In some non-limiting embodiments or aspects, the database 102 can store at least one data set including a plurality of data records. In some non-limiting embodiments or aspects, the database 102 can be in communication with the machine learning system 104, e.g., to transmit at least a portion of the data set to the machine learning system 104 for analysis and / or to receive and store data (e.g., outputs, relevance scores, loss values, etc.) from the machine learning system 104. In some non-limiting embodiments or aspects, the data set stored in the database 102 can include transaction data (e.g., real-time transaction data and / or streaming transaction data, historical transaction data, any combination thereof, etc.). In some non-limiting embodiments or aspects, the database 102 can be in communication with the querying device 106, e.g., to transmit data (e.g., outputs, relevance scores, loss values, etc.) to the querying device 106 and / or to receive and store feedback data from the querying device 106.

[0120] The machine learning system 104 can include at least one computing device, as described herein. In some non-limiting embodiments or aspects, the querying device 106 can include at least one processor (e.g., a multi-core processor), such as a graphics processing unit (GPU), a central processing unit (CPU), an accelerated processing unit (APU), a microprocessor, etc. In some non-limiting embodiments or aspects, the machine learning system 104 can include at least one machine learning model. The at least one machine learning model can include at least one neural network, at least one multi-layer perceptron (MLP), at least one deep neural network (DNN), at least one attention model, at least one self-attention model, at least one multi-head self-attention model, at least one transformer model, at least one vision transformer (ViT) model, at least one convolutional neural network (CNN), at least one tree model, etc. In some non-limiting embodiments or aspects, the machine learning model of the machine learning system 104 can be trained based on a plurality of samples (e.g., data samples, etc.), as described herein. In some non-limiting embodiments or aspects, the machine learning system 104 can be used to generate decisions (e.g., during training and / or after training), as described herein. In some non-limiting embodiments or aspects, the machine learning system 104 can be in communication with the database 102, as described herein. In some non-limiting embodiments or aspects, the machine learning system 104 can be in communication with the querying device 106, as described herein.

[0121] The querying device 106 can include at least one computing device, as described herein. For example, the querying device 106 can include a computer (e.g., a portable computer, a non-mobile computer, and / or the like), a server (e.g., a single server), a group of servers, and / or other similar devices of a user. In some non-limiting embodiments or aspects, the querying device 106 can include at least one processor (e.g., a multi-core processor), such as a graphics processing unit (GPU), a central processing unit (CPU), an accelerated processing unit (APU), a microprocessor, and / or the like. In some non-limiting embodiments or aspects, the querying device 106 can include a memory, one or more storage components, one or more input components, one or more output components, and / or one or more communication interfaces, as described herein.

[0122] Figure 1 The number and arrangement of systems and / or devices shown in Figure 1 may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or arranged in a different manner than those shown in Figure 1 two or more systems or devices shown in Figure 1 may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of the system 100 can perform one or more functions described as being performed by another group of systems or another group of devices of the system 100.

[0123] Reference is made to Figure 2A system 100 for analyzing and / or improving a machine learning model is illustrated according to some non-limiting embodiments or aspects. System 100 may include a query device 106 communicating with a machine learning system 104. The query device 106 may submit a query message to the machine learning system 104. The machine learning system 104 may process the query message and return a query response containing an answer to the query message. In some non-limiting embodiments or aspects, the machine learning system 104 may be a black-box model that returns an answer to the query message without providing a user-explanatory reason for the answer. The machine learning system 104 may be a machine learning model of the previously mentioned types (e.g., at least one neural network, at least one multilayer perceptron (MLP), at least one deep neural network (DNN), at least one attention model, at least one self-attention model, at least one multi-head self-attention model, at least one transformer model, at least one visual transformer (ViT) model, at least one convolutional neural network (CNN), at least one tree model, etc.).

[0124] The machine learning system 104 can be trained on a dataset that includes data records related to entities associated with the query (such as historical data related to entities associated with the query). The machine learning system 104 can also be trained on a dataset that includes data records related to multiple entities associated with the query (such as historical data related to entities associated with the query). The data may include multivariate time series data associated with entities, which includes time-ordered data elements for multiple variables. The multivariate time series data may be unlabeled and / or labeled data, and the machine learning system 104 may be an unsupervised machine learning model and / or a supervised machine learning model.

[0125] Machine learning system 104 can generate an answer as a response to a query message based on the data on which it was trained and on parameters in the query message. Active learning system 108 can be used to train and / or improve machine learning system 104. Active learning system 108 can be a component of machine learning system 104 or a separate system from machine learning system 104. Active learning system 108 can continuously or periodically train and / or improve machine learning system 104.

[0126] Given an entity set E = {E1, E2, ..., E...} n}, where each entity represents multivariate time series data (e.g., E i =[x i1 , x i2 ,…,x it The machine learning system 104 (decision system (D)) can generate decision d for each entity and timestamp.it The machine learning system 104 (and / or the active learning system 108) can also generate a set of users U = {U1, U2,..., U n} and interest labels (Y), where the purpose of the active learning system 108 can be to learn a function : E, D → Y that approximates the expertise of users as a timestamp of an entity.

[0127] The active learning system 108 can use multiple machine learning models to train and / or improve the machine learning system 104. The active learning system 108 can be trained using a data set that includes data records related to multiple entities associated with a query (or a portion thereof). The active learning system 108 can be trained or retrained on different or additional data, such as feedback data described below.

[0128] With continued reference to Figure 2 , the active learning system 108 can analyze data to train and / or improve the machine learning system 104 by receiving a data set that includes data records related to entities. The active learning system 108 can also receive query messages and corresponding query responses in order to evaluate the performance of the machine learning system 104 in answering queries and to improve future queries answered by the machine learning system 104. The active learning system 108 can receive query messages and corresponding query responses continuously, such as in real-time relative to when the query responses are returned, and / or the active learning system 108 can receive query messages and corresponding query responses periodically. The active learning system 108 can receive all query messages and corresponding query responses processed by the machine learning system 104. The active learning system 108 can receive a subset of all query messages and corresponding query responses processed by the machine learning system 104, such as a subset of query messages and corresponding query responses for which a confidence score of the machine learning system 104 does not satisfy a threshold.

[0129] With continued reference to Figure 2 , the active learning system 108 can include multiple machine learning models having an entity embedder 110, a user embedder 112, a feature generator 114, and a recommender 116. In Figure 2In the non-limiting example of FIG. 1, the output of the entity embedder 110 can be the input of the user embedder 112, the output of the user embedder 112 can be the input of the feature generator 114, and the output of the feature generator 114 can be the input of the recommender 116. However, it should be appreciated that the output of any machine learning model of the active learning system 108 can be the input of any other machine learning model of the active learning system 108. Moreover, multiple outputs of different machine learning models can be the input of another machine learning model. As one non-limiting example, the output of one or more of the entity embedder 110, the user embedder 112, and the feature generator 114 can be a direct input of the recommender 116.

[0130] Moreover, the output of any machine learning model of the active learning system 108 can be a direct or indirect input of another machine learning model of the active learning system 108. By “direct input,” it is meant that the output of a first machine learning model is input to a second machine learning model without first substantially changing the data (e.g., excluding non-substantial formatting changes). By “indirect input,” it is meant that the output of a first machine learning model is input to a second machine learning model after the output has undergone a substantial processing change from some other component. An example of an indirect input would be that the output of a first machine learning model is directly input to a third machine learning model, and the output of the third machine learning model is directly input to the second machine learning model.

[0131] With continued reference to Figure 2 , the active learning system 108 can analyze data to train and / or improve the machine learning system 104 by the active learning system 108 receiving a data set comprising data records associated with entities and / or historical query messages and corresponding query responses and inputting the data to the entity embedder 110. The entity embedder 110 can be pre-trained on this data in order to analyze subsequent query message-query response pairs. In response to receiving a query message-query response pair to be analyzed by the active learning system 108, the pre-trained entity embedder 110 can automatically generate a plurality of first embedding vectors (also referred to as “entity embedding vectors”) associated with entities associated with the query message-query response pair. The entity embedder 110 can comprise an unsupervised multivariate time series embedder to represent time series data of entities.

[0132] The output of the entity embedder 110 can be an embedding h it of the entity E it :

[0133] h it = S([x i(t-τ) ; …;x it ])(1)

[0134] where S is an unsupervised multivariate time series embedder, x it is a multivariate time series data of E it , t is the last timestamp, and [·; ·] is a row concatenation operator.

[0135] Referring to Figure 3 , a schematic diagram of a system for generating entity embeddings is shown in accordance with some non-limiting embodiments or aspects. As shown, Figure 3 entity multivariate time series data 122 can be input to entity embedder 110. In response to receiving multivariate time series data 122, entity embedder 110 can automatically generate entity embeddings 126 in an entity embedding space 124 (e.g., an h-dimensional space).

[0136] Referring again to Figure 2 , entity embedder 110 can input (directly or indirectly) entity embedding vectors to user embedder 112. In response, user embedder 112 can automatically generate second embedding vectors (also referred to as “user embedding vectors”) by mapping each respective entity embedding vector of the plurality of entity embedding vectors to a respective user embedding vector in a user-specific embedding space associated with a user. As described herein, a user can correspond to a user that provides feedback data to active learning system 108.

[0137] Entity embedding space 124 (from Figure 3 ) can not represent the expertise of a user of the plurality of users, whereas a user-specific embedding space can be used to distinguish between entities that are relevant or irrelevant to the user.

[0138] The output of user embedder 112 can be a user embedding for user A according to equation (2) :

[0139] = g A (h it )(2)

[0140] where g is a user embedding mapper, where g can be designed for any function such as an identity network or a neural network.

[0141] Referring to Figures 4-5 , a schematic diagram of a system for generating user embeddings is shown in accordance with some non-limiting embodiments or aspects. As shown, Figures 4-5 output from entity embedder 110 (e.g., entity embeddings 126 in entity embedding space 124) can be input to user embedder 112. User embedder 112 can automatically generate user embeddings 130 in a user embedding space 128 (for user A in this particular example).

[0142] In Figures 4-5 particular, the benefits of the user embedder 112 are further illustrated in the case where the active learning system 108 is applied to an anomaly detection problem. The user embedding space 128 can map embedding vectors that are more relevant to a user (e.g., user A) to be in close proximity to each other. In the entity embedding space 124, the light-colored embedding vector (light-colored X) can be the vector that is most relevant to user A, but the entity embedding space 124 can not be designed to group those vectors that are most relevant to user A in close proximity to each other, but the application of the user embedder 112 can cause those more relevant embedding vectors to be clustered closer to each other.

[0143] Figure 5 Similar to Figure 4 but providing a labeled non-limiting example of clustering. The entity embedding space 124 can group related entities together. For example, the entities can be merchants involved in a payment transaction, types of anomalous behavior, etc. Figure 5 A non-limiting example of has an entity embedding space 124 that groups entities together based on the type of anomalous behavior (e.g., data issues, authentication related issues, seasonal changes, etc.). While the entity embedding space 124 can be configured to group related entities together, the user can be trained on different entities and / or unrelated entities. Thus, the user embedder 112 can re-group the embedding vectors (e.g., as user embedding vectors) in the user embedding space 128. Figure 5 It is shown how the light-colored embedding vector (light-colored X) is now grouped by the user embedder 112 in the user embedding space 128 as a relevant embedding vector, forming a cluster that is as relevant as within the area of expertise of user A.

[0144] In some non-limiting embodiments or aspects, the user embedder 112 can include a user embedding mapper layer. The user embedding mapper layer can include at least one of: an identity layer, a linear layer, a non-linear layer, a non-linear layer 2 (2 non-linear layers), or any combination thereof. The identity layer can return the same embedding space, while the non-linear layer can use a sigmoid activation. The user embedding mapper layer can include a linear user embedding mapper layer.

[0145] Referring again to Figure 2 , the user embedder 112 can input the user embedding vectors (directly or indirectly) to the feature generator 114. In response, the feature generator 114 can automatically extract feature data associated with at least one feature based on the plurality of user embedding vectors.

[0146] In some non-limiting embodiments or aspects, the active learning system 108 can not have training data associated with the user (e.g., by an initial survey of the user, etc.). The feature generator 114 can extract features from the user embedding space, and these features can be learned or designed for a particular application. As one non-limiting example, an expert-designed feature for an anomaly application can be the average distance from one item to other items, which would likely be higher for an anomaly case. However, learned features can be shown to be more expressive than expert-designed features, as there can be challenges in designing or engineering all useful features. Thus, the feature generator 114 can be a functional layer on top of the user embedder 112, such that it will extract new features from learned features.

[0147] The output of the feature generator 114 can be feature data for the user A according to Equation (3) :

[0148] = f( )(3)

[0149] where h’ has a smaller dimension than h, and f is a feature extractor function. In some non-limiting embodiments or aspects, f can also represent a set of learned or expert-designed functions, in which case h’ can be a concatenation of the extracted features.

[0150] With continued reference to Figure 2 , the feature generator 114 can input the feature data (directly or indirectly) to the recommender 116 to generate a relevance score for each of the plurality of entities. In some non-limiting embodiments or aspects, at least one of: the output from the machine learning system 104 (e.g., d it ), the plurality of entity embedding vectors (e.g., h it ), the plurality of user embedding vectors (e.g., ), the feature data (e.g., ), or any combination thereof, can be input (directly or indirectly) to the recommender 116 to generate the relevance score for each of the plurality of entities.

[0151] In some non-limiting embodiments or aspects, the plurality of user embedding vectors and the feature data can be input to the recommender 116 to generate the relevance score for each of the plurality of entities.

[0152] In some non-limiting embodiments or aspects, the output from the machine learning system 104 and the feature data can be input to the recommender 116 to generate the relevance score for each of the plurality of entities.

[0153] In some non-limiting embodiments or aspects, the output of the machine learning system 104 can be input to the recommender 116 to generate a relevance score for each entity of the plurality of entities.

[0154] In some non-limiting embodiments or aspects, the plurality of entity embedding vectors can be input to the recommender 116 to generate a relevance score for each entity of the plurality of entities.

[0155] In some non-limiting embodiments or aspects, the plurality of user embedding vectors can be input to the recommender 116 to generate a relevance score for each entity of the plurality of entities.

[0156] In some non-limiting embodiments or aspects, the feature data can be input to the recommender 116 to generate a relevance score for each entity of the plurality of entities.

[0157] In response to receiving the input, the recommender 116 can automatically generate a relevance score for each entity of the plurality of entities. The relevance score can quantify the relevance of each entity to the user (e.g., user A).

[0158] In some non-limiting embodiments or aspects, the output of the recommender 116 can be a relevance score for entity E it of user A at time t according to equation (4) :

[0159] = w1ʘ d it + Σ W ʘ (4)

[0160] where the relevance score is for entity i to user A at time t, where w1and W are learned weights. These weights can represent the importance of the output of the decision system (e.g., the machine learning system 104) and the extracted features to different users.

[0161] In some non-limiting embodiments or aspects, the output of the machine learning system 104 (d it ) can be used to determine the relevance score. Even though d it may not be completely correct, it can be incorporated into the relevance score calculation to mitigate the cold start problem.

[0162] Based on the relevance scores generated by the recommender 116, the recommender can determine a subset of entities Q. The entities can include entities for which the relevance scores indicate the highest relevance to the user (e.g., user A). For example, Q can contain a predetermined number of entities (e.g., the 10 most relevant entities). For example, Q can contain all entities for which the relevance scores satisfy a predetermined relevance threshold (e.g., entities for which the relevance scores are the highest or lowest).

[0163] Each user can include a capacity representing a number of entities that the user is able to provide feedback for over a given time period. The number of entities in the subset Q can be based on the capacity of the user, such as the number of entities in the subset Q being equal to or less than the capacity of the user during the time period.

[0164] With continued reference to Figure 2 , the recommender 116 can generate a feedback request containing a subset of entities. The recommender 116 can transmit the feedback request to a user device 118 of a user. For example, the user device 118 can be associated with user A, and the feedback request can contain a subset of entities determined by the recommender 116 to be most relevant to user A.

[0165] With reference to Figure 2 and Figure 6 , a system transmitting feedback requests to user devices 118 (118a, 118b, 118c) is shown. The recommender 116 can transmit a first feedback request to the user A device 118a containing a first entity (El) determined by the recommender 116 to be relevant to user A. The recommender 116 can transmit a second feedback request to the user B device 118b containing a second entity (E2) determined by the recommender 116 to be relevant to user B. The recommender 116 can transmit a third feedback request to the user C device 118c containing a third entity (E3) determined by the recommender 116 to be relevant to user C.

[0166] With reference to Figure 2 , in response to receiving the feedback request, the user device 118 can generate feedback data. To generate the feedback data, the user device 118 can display at least one user interface associated with the entity contained in the feedback request for which feedback data is requested.

[0167] In some non-limiting embodiments or aspects, the user device 118 can display a first user interface configured to receive data from a user associated with whether the user agrees with a decision of the machine learning system 104 for an entity. The user can input an answer (e.g., a “yes” or “no” indicator) as to whether the machine learning system 104 made a correct decision for the entity. The user can additionally or alternatively provide an answer that the user believes should have been made for the entity, which can then be compared to the answer from the machine learning system 104 to determine whether the user agrees with the machine learning system 104.

[0168] In some non-limiting embodiments or aspects, the user device 118 can display a second user interface (the same or separate from the first user interface) configured to receive data from the user associated with whether the user has expertise in this entity. For example, the user can input a binary answer, such as the user does or does not have expertise in this entity. Additionally or alternatively, the user can provide a non-binary answer specifying a degree of expertise the user has in the entity (e.g., a 1-10 scale of how strong the user’s expertise is in the entity).

[0169] Accordingly, the user device 118 can receive input from the user regarding what the user indicates the decision should be (and / or whether the decision is consistent with the machine learning system 104) for the entity (“Type 1 data”) and the user’s level of expertise in the entity (“Type 2 data”). These responses can include a binary classification of the user’s responses and / or a non-binary classification of the user’s responses. Similar data can be received from the user for each entity included in the feedback request.

[0170] The user device 118 can generate feedback data based on the user’s input to the user interface. The user device 118 can transmit this feedback data to the database 102, and the database 102 can store this feedback data. The label generator 120 can generate labels and associate these labels with the feedback data before the feedback data is transmitted to the active learning system 108 and / or the machine learning system 104. The labels can be configured to enable the active learning system 108 and / or the machine learning system 104 to learn from the generated feedback data.

[0171] With continued reference to Figure 2 The feedback data can be transmitted to the active learning system 108 and / or the machine learning system 104 to train and / or improve them. At least one of the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof can be trained based on the feedback data, e.g., by inputting the feedback data thereto.

[0172] In some non-limiting embodiments or aspects, training at least one of the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof can include modifying at least one parameter of at least one of the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof based on the feedback data.

[0173] For example, Type 1 data from the feedback data can be input to the machine learning system 104 to train and / or improve the machine learning system 104, which can learn improved decisions based on the expert user’s answers about what decisions should have been made in the feedback data.

[0174] For example, Type 2 data from the feedback data can be input to at least one of the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof to train and / or improve the models, which can learn improved decisions about which user device to submit a feedback request based on the expert user’s indications of their expertise for entities in the feedback data.

[0175] In some non-limiting embodiments or aspects, training at least one of the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof can include (e.g., by the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, or the recommender 116, or some other component) automatically determining a loss based on the feedback data and a loss function. The at least one of the machine learning system 104, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof can be trained based on the loss.

[0176] In some non-limiting embodiments or aspects, the loss function used to determine the loss can include at least one of: a first component to widen a gap between pairs of entities in the entity subset that have feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the entity subset that have feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the entity subset that have feedback data associated with incorrect relative rankings between the entities in the entity subset and other entities in the plurality of entities that are not included in the entity subset; or any combination thereof.

[0177] For example, the first component can widen a gap between correctly ranked positive samples and negative samples (e.g., where the user feedback data matches the decisions of the machine learning system 104 and / or the recommender 116).

[0178] For example, the second component can narrow the gap between incorrectly ranked positive samples and negative samples (e.g., where user feedback data does not match the decisions of the machine learning system 104 and / or the recommender 116).

[0179] For example, the third component can recover un-recommended samples by narrowing the gap between incorrectly recommended samples and un-recommended samples (e.g., where user feedback data indicates that a sample should not have been recommended and there are other samples that were not recommended by the recommender 116).

[0180] It should be appreciated that these three components are for example purposes only, and more or fewer components can be used in the loss function.

[0181] In some non-limiting embodiments or aspects, the loss function can be determined as follows. For each timestamp t, the system can train with the goal to improve the relevance score of the entity = [ ,…, ] based on the array of expert knowledge information = [ ,…, ]. The loss function can be expressed according to equation (5):

[0182] maxL ALL ( ) =

[0183] x1* →L WIDEN ( component 1)

[0184] =1 =0

[0185] + x2* →L NARROW ( component 2)

[0186] =1 =0

[0187] + x3* →L RECOVER ( component 3)

[0188] =1 (5)

[0190] where L WIDEN widens the gap between correctly ranked positive samples and negative samples, L NARROW narrows the gap between incorrectly ranked positive samples and negative samples, and L RECOVERUnrecommended samples are recovered by narrowing the gap between incorrectly recommended samples and unrecommended samples. It is the relevance score of user A at time t, generated by an algorithm. It is a binary professional knowledge information array of Q entities from user A at time t. Parameters x1, x2, and x3 are adjustable hyperparameters that can be optimized based on the application scenario and requirements to evaluate each different loss function term for each component. It is an activation function.

[0191] This can be achieved by maximizing the loss function (e.g., maxL). ALL ( To train at least one of the following: machine learning system 104, entity embedder 110, user embedder 112, feature generator 114, recommender 116, or any combination thereof.

[0192] refer to Figure 7 An example system 100 is shown for analyzing and / or improving machine learning models used in payment networks 132. In addition... Figure 7 In addition to providing a non-limiting example of a query device 106 including a payment network 132, Figure 2 and Figure 7 Same. However, it should be understood that, Figure 2 The application of System 100 in the system is not limited to Figure 7 The specific query device 106 shown is (e.g., payment network 132), and the system can be used for any other suitable application.

[0193] Continue to refer to Figure 7 And refer to Figure 8 The query device 106 may include a payment network 132. The payment network 132 may include a payment device 134, a merchant system 136, a transaction service provider system 138, and an issuer system 140. The payment network 132 may process electronic payment transactions as follows.

[0194] A consumer can use a payment device 134 to initiate an electronic payment transaction with a merchant system 136 of a merchant. The merchant system 136 can generate and transmit a transaction request to a transaction service provider system 138 (of a transaction service provider associated with the payment device 134), and the transaction request can include payment device data (e.g., PAN, expiration date, cvv code, etc.) and transaction data (e.g., price, merchant identifier, merchant category code, etc.). The transaction service provider system 138 can generate and transmit an authorization request to an issuer system 140 (of an issuer system associated with the payment device 134), and the authorization request can include at least a portion of the data included in the transaction request. The issuer system 140 can generate an authorization decision for the electronic payment transaction, such as the transaction being authorized, partially authorized, or declined. The issuer system 140 can generate and transmit an authorization response to the transaction service provider system 138, which includes the authorization decision. The transaction service provider system 138 can generate and transmit a transaction response to the merchant system 136, which includes the authorization decision. The merchant system 136 can automatically initiate further processing of the electronic payment transaction based on the authorization decision. For example, if the authorization decision is to decline the transaction, the merchant system 136 can automatically terminate the transaction. For example, if the authorization decision is to decline the transaction, the merchant system 136 can automatically initiate processing of the payment transaction until completion. The payment network 132 can authorize, clear, and settle the transaction. The payment network 132 can store transaction data (also referred to herein as “historical transaction data”) associated with electronic payment transactions processed thereby. The historical transaction data can include multivariate time series data associated with electronic payment transactions.

[0195] With continued reference to Figure 7 In some non-limiting embodiments or aspects, the payment network 132 can send a query request to the machine learning system 104 to request at least one of: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, other types of transaction-related decisions, or any combination thereof. The machine learning system 104 can generate at least one of the foregoing decisions by analyzing historical transaction data (and / or historical credit application data or historical insurance application data). The machine learning system 104 can return a query response to the payment network 132 that includes its decision. The payment network 132 can further process the transaction based on the query response.

[0196] The machine learning system 104 can be trained using the active learning system 108 as previously described herein. The machine learning system 104 can be continuously trained as further historical transaction data is generated by the payment network 132 and / or feedback data from the active learning system 108, and / or the machine learning system 104 can be periodically trained on further historical transaction data generated by the payment network 132 and / or feedback data from the active learning system 108. The machine learning system 104 and / or the active learning system 108 can reevaluate a particular transaction based on a request from the payment network 132 and / or based on a confidence score generated by the machine learning system 104 for the particular payment transaction that does not satisfy a threshold using the user device 118 as previously described herein.

[0197] With continued reference to Figure 7 , the active learning system 108 can train the machine learning system 104 for transactions processed by the payment network 132 using a dataset comprising a plurality of data records associated with entities of the plurality of entities, the plurality of data records comprising historical transaction data generated and / or processed by the payment network 132. Accordingly, the historical transaction data can be input to at least one of the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, or any combination thereof of the active learning system 108.

[0198] Reference is now made to Figure 9 , a process 200 for analyzing and / or improving a machine learning model is shown, in accordance with some non-limiting embodiments or aspects. Figure 9 The steps shown are for example purposes only. It will be appreciated that, in some non-limiting embodiments or aspects, additional, fewer, different, and / or a different order of steps can be used. In some non-limiting embodiments or aspects, a step can be performed automatically in response to the performance and / or completion of a previous step. In some non-limiting embodiments or aspects, one or more of the steps of the process 200 can be performed (e.g., entirely, partially, etc.) by the active learning 108 (e.g., one or more devices of the active learning system 108). In some non-limiting embodiments or aspects, one or more of the steps of the process 200 can be performed (e.g., entirely, partially, etc.) by another system, another device, another group of systems, or another group of devices separate from or including the active learning system 108.

[0199] As Figure 9 shown, at step 202, the process 200 can include receiving a dataset. For example, the active learning system 108 can receive a dataset comprising a plurality of data records, each data record of the plurality of data records being associated with an entity of a plurality of entities.

[0200] In some non-limiting embodiments or aspects, the dataset may include multivariate time series data with multiple data records, each of which includes time-ordered data elements for multiple variables.

[0201] In some non-limiting implementations or aspects, the dataset may include transaction data associated with multiple transactions, with each data record associated with at least one of the multiple transactions.

[0202] like Figure 9 As shown, in step 204, process 200 may include inputting a dataset into a pre-trained first machine learning model. For example, active learning system 108 may input the dataset into a pre-trained first machine learning model (e.g., entity embedder 110) to generate a plurality of first embedding vectors, each of the plurality of first embedding vectors being associated with a corresponding entity among a plurality of entities.

[0203] like Figure 9 As shown, in step 206, process 200 may include inputting a first embedding vector into a second machine learning model. For example, an active learning system 108 (e.g., its entity embedder 110) may input multiple first embedding vectors into a second machine learning model (e.g., a user embedder 112) to generate multiple second embedding vectors by mapping each of the multiple first embedding vectors to a corresponding second embedding vector in a user-specific embedding space associated with the user.

[0204] like Figure 9 As shown, in step 208, process 200 may include inputting a second embedding vector into a third machine learning model. For example, an active learning system 108 (e.g., its entity embedder 110 and / or user embedder 112) may input multiple second embedding vectors into a third machine learning model (e.g., a feature generator 114) to extract feature data associated with at least one feature based on the multiple second embedding vectors.

[0205] like Figure 9 As shown, in step 210, process 200 may include inputting second embedding vectors and / or feature data into a fourth machine learning model. For example, an active learning system 108 (e.g., its entity embedder 110 and / or user embedder 112 and / or feature generator 114) may input multiple second embedding vectors and feature data into a fourth machine learning model (e.g., a recommender 116) to generate a relevance score for each of multiple entities.

[0206] In some non-restrictive implementations or aspects, the relevance score may be associated with at least one of the following: authorization decision, fraud determination, credit application decision, insurance application decision, or any combination thereof.

[0207] like Figure 9 As shown, in step 212, process 200 may include determining a subset of entities. For example, active learning system 108 may determine the subset of entities based on the relevance score of each entity in a subset of multiple entities.

[0208] In some non-limiting embodiments or aspects, the entity subset may include a predetermined number of entities having one of the highest or lowest relevance scores.

[0209] like Figure 9 As shown, in step 214, process 200 may include transmitting a feedback request. For example, active learning system 108 may transmit a feedback request to a user's user device (e.g., user device 118) based on a subset of entities.

[0210] like Figure 9 As shown, in step 216, process 200 may include receiving feedback data. For example, active learning system 108 may receive feedback data for each entity in a subset of entities from a user device (e.g., directly from user device 118 and / or indirectly via another component, such as tag generator 120).

[0211] In some non-limiting implementations or aspects, the feedback data may include a binary classification for each entity in a subset of entities.

[0212] like Figure 9 As shown, in step 218, process 200 may include training at least one of the machine learning models based on feedback data. For example, active learning system 108 may train at least one of the following based on feedback data: machine learning system 104, entity embedder 110, user embedder 112, feature generator 114, recommender 116, or any combination thereof.

[0213] In some non-limiting embodiments or aspects, training a machine learning model may include modifying at least one parameter of at least one of the machine learning models based on feedback data. At least one parameter may include (e.g., for a neural network) at least one weight, (e.g., for a neural network) at least one element of a weight matrix, etc.

[0214] In some non-limiting embodiments or aspects, training the machine learning model can include determining, with the at least one processor, a loss based on the feedback data and a loss function, and training, with the at least one processor, at least one of the machine learning models based on the loss. The loss function can include at least one of: a first component to widen a gap between pairs of entities in the entity subset having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow the gap between pairs of entities in the entity subset having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow the gap between entities in the entity subset having feedback data associated with incorrect relative rankings between the entities in the entity subset and other entities in the plurality of entities not included in the entity subset; or any combination thereof. The loss function can further include a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

[0215] Reference is now made to Figure 10 , Figure 10 is a diagram of a non-limiting embodiment or aspect of an example environment 300 in which systems, products, and / or methods as described herein can be implemented. As shown in Figure 10 environment 300 includes a transaction service provider system 302, an issuer system 304, a customer device 306, a merchant system 308, an acquirer system 310, and a communication network 312. In some non-limiting embodiments or aspects, at least one of the database 102, the machine learning system 104, the query device 106, the active learning system 108, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, the user device 118, the label generator 120, or any combination thereof can be implemented by the transaction service provider system 302 (e.g., as part of). In some non-limiting embodiments or aspects, at least one of the database 102, the machine learning system 104, the query device 106, the active learning system 108, the entity embedder 110, the user embedder 112, the feature generator 114, the recommender 116, the user device 118, the label generator 120, or any combination thereof can be implemented by another system, another device, another group of systems, or another group of devices separate from or including the transaction service provider system 302 (such as the issuer system 304, the customer device 306, the merchant system 308, the acquirer system 310, etc.) (e.g., as part of).

[0216] Transaction service provider system 302 can include one or more devices capable of receiving information from and / or transmitting information to transaction service provider system 302, customer device 306, merchant system 308, and / or acquirer system 310 via communication network 312. For example, transaction service provider system 302 can include a computing device, such as a server (e.g., a transaction processing server, etc.), a group of servers, and / or other similar devices. In some non-limiting embodiments or aspects, transaction service provider system 302 can be associated with a transaction service provider described herein. In some non-limiting embodiments or aspects, transaction service provider system 302 can be in communication with a data storage device, which can be local or remote to transaction service provider system 302. In some non-limiting embodiments or aspects, transaction service provider system 302 can be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device.

[0217] Issuer system 304 can include one or more devices capable of receiving information from and / or transmitting information to transaction service provider system 302, customer device 306, merchant system 308, and / or acquirer system 310 via communication network 312. For example, issuer system 304 can include a computing device, such as a server, a group of servers, and / or other similar devices. In some non-limiting embodiments or aspects, issuer system 304 can be associated with an issuer institution described herein. For example, issuer system 304 can be associated with an issuer institution that issues a credit account, a debit account, a credit card, a debit card, etc. to a user associated with customer device 306.

[0218] The client devices 306 can include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the merchant system 308, and / or the acquirer system 310 via the communication network 312. Additionally or alternatively, each client device 306 can include a device capable of receiving information from and / or transmitting information to other client devices 306 via the communication network 312, another network (e.g., a temporary network, a local network, a private network, a virtual private network, etc.), and / or any other suitable communication technique. For example, the client devices 306 can include client devices, etc. In some non-limiting embodiments or aspects, the client devices 306 can or can not be capable of receiving information from the merchant system 308 or from another client device 306 via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, etc.) and / or transmitting information via a short-range wireless communication connection (e.g., to the merchant system 308).

[0219] The merchant system 308 can include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 302, the issuer system 304, the client devices 306, and / or the acquirer system 310 via the communication network 312. The merchant system 308 can also include a device capable of receiving information from and / or transmitting information to the client devices 306 via the communication network 312, a communication connection with the client devices 306 (e.g., an NFC communication connection, an RFID communication connection, a Bluetooth® communication connection, a Zigbee® communication connection, etc.), etc. In some non-limiting embodiments or aspects, the merchant system 308 can include a computing device, such as a server, a group of servers, a client device, a group of client devices, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 308 can be associated with a merchant described herein. In some non-limiting embodiments or aspects, the merchant system 308 can include one or more client devices. For example, the merchant system 308 can include a client device that allows a merchant to transmit information to the transaction service provider system 302. In some non-limiting embodiments or aspects, the merchant system 308 can include one or more devices capable of being used by a merchant to conduct transactions with users, such as a computer, a computer system, and / or a peripheral device. For example, the merchant system 308 can include a POS device and / or a POS system.

[0220] Acquiring system 310 may include one or more devices capable of receiving and / or transmitting information to transaction service provider system 302, issuer system 304, client device 306, and / or merchant system 308 via communication network 312. For example, acquiring system 310 may include computing devices, servers, server clusters, etc. In some non-limiting embodiments or aspects, acquiring system 310 may be associated with the acquiring party described herein.

[0221] The communication network 312 may include one or more wired and / or wireless networks. For example, the communication network 312 may include cellular networks (e.g., Long Term Evolution (LTE®) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMN), local area networks (LAN), wide area networks (WAN), metropolitan area networks (MAN), telephone networks (e.g., public switched telephone network (PSTN)), private networks (e.g., private networks associated with transaction service providers), ad hoc networks, intranets, the Internet, fiber-optic networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0222] Provided as an example Figure 10 The number and arrangement of the systems, devices, and / or networks shown. Figure 10 Compared to those shown, there may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or systems, devices, and / or networks arranged differently. Furthermore, implementation may be within a single system and / or device. Figure 10 Two or more systems or devices shown in the document, or Figure 10 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively or additionally, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of environment 300 may perform one or more functions described as being performed by another group of systems or another group of devices of environment 300.

[0223] Now for reference Figure 11 The diagram illustrates example components of device 400 according to a non-limiting embodiment or aspect. As an example, device 400 may correspond to at least one of the following: Figure 1 The database 102, machine learning system 104, and / or query device 106 are included; and / or Figure 2The database 102, machine learning system 104 and / or query device 106, active learning system 108, entity embedder 110, user embedder 112, feature generator 114, recommender 116, user device 118 and / or tag generator 120; and / or Figure 3 The entity embedder 110 in; and / or Figure 4 User embedder 112 in; and / or Figure 5 User embedder 112 in; and / or Figure 6 The recommender 116 and / or user equipment 118a-c; and / or Figure 7 The database 102, machine learning system 104, query device / payment network 106 / 132, active learning system 108, entity embedder 110, user embedder 112, feature generator 114, recommender 116, user device 118 and / or tag generator 120; and / or Figure 8 The payment network 132, payment device 134, merchant system 136, transaction service provider system 138 and / or issuer system 140; and / or Figure 10 The transaction service provider system 302, the issuer system 304, the client device 306, the merchant system 308, and / or the acquirer system 310; or any combination thereof. In some non-limiting embodiments or aspects, Figures 1-8 or Figure 10 Such systems or devices may include at least one device 400 and / or at least one component of device 400. Provided Figure 11 The number and arrangement of components shown are for illustrative purposes only. In some non-limiting embodiments or aspects, device 400 may include additional components, fewer components, different components, or components that are compatible with... Figure 11 The components are arranged in those different ways as shown. Alternatively, a group of components (e.g., one or more components) of device 400 may perform one or more functions described as being performed by another group of components of device 400.

[0224] like Figure 11As shown, the device 400 can include a bus 402, a processor 404, a memory 406, a storage component 408, an input component 410, an output component 412, and a communication interface 414. The bus 402 can include a component that permits communication among the components of the device 400. In some non-limiting embodiments or aspects, the processor 404 can be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 404 can include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application- specific integrated circuit (ASIC), etc.). The memory 406 can include a random access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., a flash memory, a magnetic storage device, an optical storage device, etc.) that stores information and / or instructions for use by the processor 404.

[0225] With continued reference to Figure 11 The storage component 408 can store information and / or software related to the operation and use of the device 400. For example, the storage component 408 can include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid state disk, etc.) and / or another type of computer readable media. The input component 410 can include a component that permits the device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, the input component 410 can include a sensor (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.) for sensing information. The output component 412 can include a component that provides output information from the device 400 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 414 can include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables the device 400 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 414 can permit the device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 414 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, etc.

[0226] Device 400 can perform one or more processes described herein. Device 400 can perform these processes based on processor 404 executing software instructions stored by computer-readable medium(s) 406 and / or 408, etc. The computer-readable medium(s) can include any non-transitory memory. The memory can include memory space located internally or externally to a single physical device or spread across multiple physical devices. The software instructions can be read into the memory 406 and / or storage 408 from another computer-readable medium or from another device via communication interface 414. When executed, the software instructions stored in the memory 406 and / or storage 408 can cause the processor 404 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry can be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the described embodiments are not limited to any specific combination of hardware circuitry and software. The term “configured to” as used herein can refer to an arrangement of software, devices, and / or hardware used to perform and / or implement one or more functions (e.g., actions, processes, steps of processes, etc.). For example, a “processor configured to” can refer to a processor executing software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0227] In some non-limiting embodiments or aspects, a computer program product for analyzing and / or improving a machine learning model includes at least one non-transitory computer-readable medium having program instructions that, when executed by at least one processor, cause the at least one processor to perform one of the previously described methods. The at least one processor can include Figures 1-8 or Figure 10 Any of the illustrated components (e.g., database 102, machine learning system 104, query device 106, active learning system 108, entity embedder 110, user embedder 112, feature generator 114, recommender 116, user device 118, label generator 120, payment network 132, payment device 134, merchant system 136, transaction service provider system 138, issuer system 140, transaction service provider system 302, issuer system 304, customer device 306, merchant system 308, acquirer system 310, etc.).

[0228] For purposes of illustration, the effectiveness of example systems in accordance with the present disclosure are compared to existing systems to illustrate the effectiveness of the present disclosure. A public dataset containing 38 time series of various lengths is used in these examples. The data is chunked into 100 time series (entities) of length 365. Each time series includes 38 different features.

[0229] The following systems were used in these examples:

[0230] 1. Random (baseline)—randomly recommend entities to users in the recommender step.

[0231] 2. AI decision system (baseline)—use an artificial intelligence (AI) system probability to recommend entities to users.

[0232] 3. Random forest active learner (baseline)—combine the uncertainty and confidence scores for each entity and recommend them to users. This model was trained with active learning using the same experimental setup as the feedback loop active learner (FLAL).

[0233] 4. Feedback loop active learner (FLAL) (according to the present disclosure)—a system described by the present disclosure, such as the system embodied in Figure 2 .

[0234] For the FLAL, as a result of the ablation study described below, the user embedding mapper was chosen to be a linear layer. The feature generator utilized a linear layer to generate 15 learned features, and the recommender also used a linear layer to generate the relevance scores for the entities. To simulate user feedback, the ground truth information for the dataset was used. The Adam optimizer was used with a learning rate of 0.0001 to tune the hyperparameters to optimize the model, with L2 regularization weight of 0.001 on the model weights. The loss function weights x1, x2, and x3 were chosen from {0.0, 0.5, 1.0}. In these examples, τ was set to 127, and the embedding size d was 128. For different runs, Q was set to 10 and 20. The number of recommended items was also set to Q. Since the ground truth evaluation can only consider feedback from the recommended entities, the example compared the precision metric. At each round, the precision at Q and the average precision at Q were computed.

[0235] Figures 12A-12B The precision at Q and the average precision at Q are shown for Q = 10 and Q = 20. The cumulative average of the precision performance was computed at each step. Note that since the user’s interest / expertise label was used as the ground truth decision, this performance reflects the improved AI decision performance. From Figures 12A-12B It can be seen that the FLAL outperforms the competing baselines in all reported metrics and especially after 10-20 steps.

[0236] Another consideration for learning human interests is the convergence speed. The FLAL converges around step 50, which is faster than the best baseline, the random forest active learner.

[0237] From Figures 12A-12BNote that the AI decision system is not enough. The active learner mechanism greatly improves performance, however random decisions are still worse than the original AI decision system. Another significant difference between Q = 10 and Q = 20 is in the precision performance at Q. When Q = 20, the performance drops below 0.6. However, this can essentially happen because the number of anomalies in the data does not exceed 12 (e.g., 20 x 0.6) at some time. This suggests that the number of recommended entities should be optimized based on the number of anomalies at each step using the same value of the budget Q. Since leading zeros do not affect the result, the average precision is less susceptible to this problem. Their performance is close to each other for both Q.

[0238] Reference is made to Figure 13 Ablation studies on different user embedding mapper layers using precision at 10 and precision at 20 are shown. These examples compare the identity layer, linear layer, nonlinear layer, and nonlinear layer 2 (2 nonlinear layers) of the user embedding mapper layer. The identity layer returns the same embedding space, while the nonlinear layer uses a sigmoid activation. The results show that one linear layer captures enough information and as much as one nonlinear layer. On the other hand, the identity layer has gradually increasing performance for precision at 10, but slower convergence rate. Nonlinear layer 2 has the worst performance, which can be due to overfitting from lack of data points.

[0239] Reference is made to Figure 14 Loss function term sensitivity analysis is shown. Each block represents the last step cumulative average of precision at 10 from FLAL parameterized by different x1, x2, and x3 values. The results show that all terms contribute to performance. The absence of the first term (e.g., x1 = 0) makes the model perform worse. The second term (x2) is not as effective as the first term, but increasing its value still makes performance better. The third term (x3) has a similar effect as the second term. They all aim to shrink the positive and negative sample rankings. The best performance is achieved when the hyperparameter area equals 1 or x1 = 1, x2 = 0.5, and x3 = 0.5.

[0240] While embodiments have been described in detail, it should be understood that the description is for illustrative purposes only and that modifications and equivalents can be used without departing from the spirit and scope of the disclosure. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment can be combined with one or more features of any other embodiment.

Claims

1. A computer-implemented method for processing payment transactions at a payment network, the computer-implemented method comprising: receiving, with at least one processor, a data set comprising a plurality of data records, the data set comprising transaction data associated with a plurality of payment transactions processed by a payment network, each data record associated with a payment transaction of the plurality of payment transactions, and each data record associated with an entity of a plurality of entities, the entity being a merchant in the payment transaction or a type of abnormal behavior, the type of abnormal behavior comprising a data issue, an authentication issue, or a seasonal change; inputting, with at least one processor, the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors associated with an entity of the plurality of entities and a query message-query response pair; inputting, with at least one processor, the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; inputting, with at least one processor, the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; inputting, with at least one processor, the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; the relevance score associated with at least one of the following decisions: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof; determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities; receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device; and training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, wherein the feedback data improves an accuracy of the decisions produced by the machine learning models, the machine learning system returns a query response comprising the decisions to the payment network, the payment network processes the payment transactions based on the query response.

2. The computer-implemented method of claim 1, wherein the at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, is trained, including: modifying at least one parameter of the at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

3. The computer-implemented method of claim 1, wherein the feedback data comprises a binary classification for each entity of the subset of entities.

4. The computer-implemented method of claim 1, wherein the dataset comprises multivariate time series data having the plurality of data records, each respective data record of the plurality of data records comprising time-ordered data elements for a plurality of variables.

5. The computer-implemented method of claim 1, wherein the at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof is trained, comprising: determining, with at least one processor, a loss based on the feedback data and a loss function; and training, with at least one processor, the at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the loss.

6. The computer-implemented method of claim 5, wherein the loss function comprises at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities of each pair based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities of each pair based on their relevance scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with an incorrect relative ranking between an entity of the subset of entities and other entities of the plurality of entities not included in the subset of entities; or any combination thereof.

7. The computer-implemented method of claim 6, wherein the loss function further comprises a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

8. The computer-implemented method of claim 1, wherein the subset of entities comprises a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

9. A system for processing payment transactions by a payment network, the system comprising at least one processor configured to: receive a dataset comprising a plurality of data records, the dataset comprising transaction data associated with a plurality of payment transactions processed by a payment network, each data record associated with a payment transaction of the plurality of payment transactions, and each data record associated with an entity of a plurality of entities, the entity being a merchant or a type of abnormal behavior in the payment transaction, the type of abnormal behavior comprising a data issue, an authentication issue, or a seasonal change; input the dataset to a pre-trained first machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors associated with an entity of the plurality of entities associated with a query message-query response pair; inputting the plurality of first embedding vectors to a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with the user; inputting the plurality of second embedding vectors to a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; inputting the output from the machine learning system and the feature data to a fourth machine learning model to generate a relevance score for each entity of the plurality of entities; the relevance score associated with at least one of the following decisions: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof; determining a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; transmitting a feedback request to a user device of the user based on the subset of entities; receiving feedback data for each entity of the subset of entities from the user device; and training at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, wherein the feedback data improves an accuracy of the decisions produced by the machine learning models, the machine learning system returns a query response including the decisions to the payment network, the payment network processes the payment transaction based on the query response.

10. The system of claim 9, wherein the at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, is trained, including: modifying at least one parameter of the at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

11. The system of claim 9, wherein the feedback data includes a binary classification for each entity of the subset of entities.

12. The system of claim 9, wherein the dataset includes multivariate time series data having the plurality of data records, each respective data record of the plurality of data records including time-ordered data elements for a plurality of variables.

13. The system of claim 9, wherein the at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, is trained, including: the at least one processor is configured to: determine a loss based on the feedback data and a loss function; and train the at least one of the following based on the loss: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof.

14. The system of claim 13, wherein the loss function includes at least one of: a first component to widen a gap between pairs of entities of the subset of entities having feedback data associated with a correct relative ranking between the entities of each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities of the subset of entities having feedback data associated with an incorrect relative ranking between the entities of each pair of entities based on their relevance scores; a third component configured to shrink a gap between entities in the subset of entities having feedback data associated with incorrect relative rankings between entities in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

15. The system of claim 14, wherein the loss function further comprises a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

16. The system of claim 9, wherein the subset of entities comprises a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

17. A computer program product for processing payment transactions by a payment network, the computer program product comprising at least one non-transitory computer-readable medium including program instructions, the program instructions, when executed by at least one processor, cause the at least one processor to: receive a dataset comprising a plurality of data records, the dataset comprising transaction data associated with a plurality of payment transactions processed by a payment network, each data record associated with a payment transaction in the plurality of payment transactions, and each data record associated with an entity in a plurality of entities, the entity being a merchant in the payment transaction or a type of abnormal behavior, the type of abnormal behavior comprising a data issue, an authentication issue, or a seasonal change; input the dataset into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector in the plurality of first embedding vectors associated with an entity in the plurality of entities and associated with a query message-query response pair; input the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector in the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; input the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; input the output from the machine learning system and the feature data into a fourth machine learning model to generate a relevance score for each entity in the plurality of entities; the relevance score associated with at least one of the following decisions: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof; determine a subset of entities in the plurality of entities based on the relevance score for each entity in the subset of entities; transmit a feedback request to a user device of the user based on the subset of entities; receive feedback data for each entity in the subset of entities from the user device; and training at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the feedback data, wherein the feedback data improves accuracy of the decisions produced by the machine learning models, the machine learning system returns a query response including the decisions to the payment network, the payment network processes the payment transaction based on the query response.

18. The computer program product of claim 17, wherein the at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, is trained, including: modifying at least one parameter of the at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the feedback data.

19. The computer program product of claim 17, wherein the feedback data includes a binary classification for each entity in the subset of entities.

20. The computer program product of claim 17, wherein the dataset includes multivariate time series data having the plurality of data records, each respective data record in the plurality of data records including time-ordered data elements for a plurality of variables.

21. The computer program product of claim 17, wherein the at least one of: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, is trained, comprising: the program instructions cause the at least one processor to: determine a loss based on the feedback data and a loss function; and train the at least one of the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof based on the loss.

22. The computer program product of claim 21, wherein the loss function includes at least one of: a first component to widen a gap between pairs of entities in the subset of entities having feedback data associated with a correct relative ranking between the entities in each pair of entities based on their relevance scores; a second component configured to narrow a gap between pairs of entities in the subset of entities having feedback data associated with an incorrect relative ranking between the entities in each pair of entities based on their relevance scores; a third component configured to narrow a gap between entities in the subset of entities having feedback data associated with incorrect relative rankings between the entities in the subset of entities and other entities in the plurality of entities not included in the subset of entities; or any combination thereof.

23. The computer program product of claim 22, wherein the loss function further includes a first hyperparameter for the first component, a second hyperparameter for the second component, and a third hyperparameter for the third component.

24. The computer program product of claim 17, wherein the subset of entities includes a predetermined number of entities having one of a highest relevance score or a lowest relevance score.

25. A computer-implemented method for processing payment transactions at a payment network, the computer-implemented method comprising: ​ Receiving, with at least one processor, a data set comprising a plurality of data records, the data set comprising transaction data associated with a plurality of payment transactions processed by a payment network, each data record associated with a payment transaction of the plurality of payment transactions, and each data record associated with an entity of a plurality of entities, the entity being a merchant in the payment transaction or a type of abnormal behavior, the type of abnormal behavior comprising a data issue, an authentication issue, or a seasonal change; Inputting, with at least one processor, the data set into a first pre-trained machine learning model to generate a plurality of first embedding vectors, each respective first embedding vector of the plurality of first embedding vectors associated with an entity of the plurality of entities and a query message-query response pair associated with the entity; Inputting, with at least one processor, the plurality of first embedding vectors into a second machine learning model to generate a plurality of second embedding vectors by mapping each respective first embedding vector of the plurality of first embedding vectors to a respective second embedding vector in a user-specific embedding space associated with a user; Inputting, with at least one processor, the plurality of second embedding vectors into a third machine learning model to extract feature data associated with at least one feature based on the plurality of second embedding vectors; Inputting, with at least one processor, at least one of the following into a fourth machine learning model to generate a relevance score for each entity of the plurality of entities: an output from a machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, the feature data, or any combination thereof; the relevance score associated with at least one of the following decisions: an authorization decision, a fraud determination, a credit application decision, an insurance application decision, or any combination thereof Determining, with at least one processor, a subset of entities of the plurality of entities based on the relevance score for each entity of the subset of entities; Transmitting, with at least one processor, a feedback request to a user device of the user based on the subset of entities; Receiving, with at least one processor, feedback data for each entity of the subset of entities from the user device; And Training, with at least one processor, at least one of the following based on the feedback data: the second machine learning model, the third machine learning model, the fourth machine learning model, or any combination thereof, wherein the feedback data improves an accuracy of the decisions produced by the machine learning models, the machine learning system returns a query response comprising the decisions to the payment network, the payment network processes the payment transactions based on the query response.

26. The computer-implemented method of claim 25, wherein at least two of the output from a machine learning system, the plurality of first embedding vectors, the plurality of second embedding vectors, and the feature data are inputted into the fourth machine learning model to generate the relevance score for each entity of the plurality of entities.

27. The computer-implemented method of claim 25, wherein the output from the machine learning system is input to the fourth machine learning model to generate a relevance score for each entity of the plurality of entities.

28. The computer-implemented method of claim 25, wherein the plurality of first embedding vectors are input to the fourth machine learning model to generate a relevance score for each entity of the plurality of entities.

29. The computer-implemented method of claim 25, wherein the plurality of second embedding vectors are input to the fourth machine learning model to generate a relevance score for each entity of the plurality of entities.

30. The computer-implemented method of claim 25, wherein the feature data is input to the fourth machine learning model to generate a relevance score for each entity of the plurality of entities.

Images