Distributed execution of machine learning model on cluster of servers

Abstract

Systems, methods, and computer program products for distributed execution of a machine learning model on a cluster of servers are described. The method includes initiating retrieval of a machine learning model from a data repository and converting the machine learning model into an executable format. The method includes sending a converted machine learning model to each node of the cluster of servers and executing the converted machine learning model on each node. The method includes generating an initial performance metric based on executing the transformed machine learning model on each node. The method includes sending the plurality of initial performance metrics from each node to an external processor and combining the plurality of initial performance metrics to produce a combined performance metric. The method includes modifying a model hyper-parameter of the machine learning model based on the combined performance metric, and executing the modified machine learning model in a computer system to evaluate real-time event data.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning model execution and, in non-limiting embodiments or aspects, to systems, methods, and computer program products for distributed execution of machine learning models on a cluster of servers. Background Art

[0002] In a world that requires accurate, well-trained, and on-demand results, the deployment of machine learning models is an ever-changing challenge. Machine learning models may need to be trained with large amounts (e.g., petabytes) of historical data before being deployed in a production environment. High-performance machine learning models may need to be resilient to time-based pattern changes such as seasonality, which may increase training data requirements. When assembling machine learning models for execution in a production environment, multiple machine learning models may need to be tested to determine the best performing model. If the above process is performed by a single computer, a large amount of historical data needs to be retrieved from the storage device for training and / or execution of the machine learning model, which will consume a large amount of processing time in proportion. In addition, in a networked system, sending large amounts of historical data to the machine learning model may consume a large amount of bandwidth and will slow down the entire process from training to evaluation to execution.

[0003] Therefore, there is a need in the art for a technical solution that solves the problem of sending stored data to machine learning models for training, testing, etc., and provides a faster development to implementation workflow while also reducing computer resource requirements. Summary of the invention

[0004] According to some non-limiting embodiments or aspects, systems, methods, and computer program products for distributed execution of machine learning models on a cluster of servers are provided that overcome some or all of the deficiencies identified above.

[0005] According to some non-limiting embodiments or aspects, a computer-implemented method for distributed execution of machine learning models on a server cluster is provided. The method includes receiving, with at least one processor, a request identifying at least one machine learning model. The method also includes initiating, with at least one processor, retrieval of the at least one machine learning model from a data repository based on the request. The method also includes converting, with at least one processor, the at least one machine learning model from an initial format to an executable format compatible with distributed execution on the server cluster to produce at least one converted machine learning model. The method also includes sending, with at least one processor, the at least one converted machine learning model to each of at least two nodes of the server cluster. The method also includes executing, with at least one processor, the at least one converted machine learning model on each of the at least two nodes using data stored on the at least two nodes. The method also includes generating, with at least one processor, an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to produce a plurality of initial performance metrics. The method also includes sending, with at least one processor, the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster. The method also includes combining, with at least one processor, the plurality of initial performance metrics to produce at least one combined performance metric of the at least one converted machine learning model. The method also includes modifying, using at least one processor, at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model. The method also includes executing, using at least one processor, the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0006] In some non-limiting embodiments or aspects, the request may include at least one data parameter associated with a subset of data stored in the server cluster. Executing the at least one converted machine learning model on each of the at least two nodes may also include inputting the subset of data stored on each node to the at least one converted machine learning model based on the at least one data parameter.

[0007] In some non-limiting embodiments or aspects, converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster includes: using at least one processor to convert the at least one machine learning model from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library. The method may also include determining, using at least one processor, the new format based on a programming language used to operate the server cluster.

[0008] In some non-limiting embodiments or aspects, the initial performance metric generated at each of the at least two nodes may include false positive and false negative error rates based on the at least one converted machine learning model.

[0009] In some non-limiting embodiments or aspects, the at least one combined performance metric may include at least one of: area under receiver operating characteristic (AUROC); model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0010] In some non-limiting embodiments or aspects, the at least one model hyperparameter may include at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0011] According to some non-limiting embodiments or aspects, a system for distributed execution of machine learning models on a server cluster is provided. The system includes at least one server, the at least one server including at least one processor. The at least one server is programmed or configured to receive a request identifying at least one machine learning model. The at least one server is also programmed or configured to initiate retrieval of the at least one machine learning model from a data repository based on the request. The at least one server is also programmed or configured to convert the at least one machine learning model from an initial format to an executable format compatible with distributed execution on the server cluster to generate at least one converted machine learning model. The at least one server is also programmed or configured to send the at least one converted machine learning model to each of at least two nodes of the server cluster. The at least one server is also programmed or configured to execute the at least one converted machine learning model on each of the at least two nodes using data stored on each of the at least two nodes. The at least one server is also programmed or configured to generate initial performance metrics on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to generate a plurality of initial performance metrics. The at least one server is also programmed or configured to send the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster. The at least one server is also programmed or configured to combine the plurality of initial performance metrics to generate at least one combined performance metric of the at least one converted machine learning model. The at least one server is also programmed or configured to modify at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to generate at least one modified machine learning model. The at least one server is also programmed or configured to execute the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0012] In some non-limiting embodiments or aspects, the request may further include at least one data parameter associated with a subset of data stored in the server cluster. Executing the at least one converted machine learning model on each of the at least two nodes may further include inputting the subset of data stored on each node to the at least one converted machine learning model based on the at least one data parameter.

[0013] In some non-limiting embodiments or aspects, converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster may include converting the at least one machine learning model from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library. The at least one server may also be programmed or configured to determine the new format based on a programming language used to operate the server cluster.

[0014] In some non-limiting embodiments or aspects, the initial performance metric generated at each of the at least two nodes may include false positive and false negative error rates based on the at least one converted machine learning model.

[0015] In some non-limiting embodiments or aspects, the at least one combined performance metric may include at least one of: receiver AUROC; model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0016] In some non-limiting embodiments or aspects, the at least one model hyperparameter may include at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0017] According to some non-limiting embodiments or aspects, a computer program product for distributed execution of machine learning models on a server cluster is provided. The computer program product includes at least one non-transitory computer-readable medium, and the at least one non-transitory computer-readable medium includes program instructions. The program instructions, when executed by at least one processor, cause the at least one processor to receive a request to identify at least one machine learning model. The program instructions also cause the at least one processor to initiate retrieval of the at least one machine learning model from a data repository based on the request. The program instructions also cause the at least one processor to convert the at least one machine learning model from an initial format to an executable format compatible with distributed execution on a server cluster to generate at least one converted machine learning model. The program instructions also cause the at least one processor to send the at least one converted machine learning model to each of at least two nodes of the server cluster. The program instructions also cause the at least one processor to execute the at least one converted machine learning model on each of the at least two nodes using data stored on each of the at least two nodes. The program instructions also cause the at least one processor to generate initial performance metrics on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to generate a plurality of initial performance metrics. The program instructions also cause the at least one processor to send the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster. The program instructions also cause the at least one processor to combine the plurality of initial performance metrics to generate at least one combined performance metric of the at least one converted machine learning model. The program instructions also cause the at least one processor to modify at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to generate at least one modified machine learning model. The program instructions also cause the at least one processor to execute the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0018] In some non-limiting embodiments or aspects, the request may further include at least one data parameter associated with a subset of data stored in the server cluster. Executing the at least one converted machine learning model on each of the at least two nodes may further include inputting the subset of data stored on each node to the at least one converted machine learning model based on the at least one data parameter.

[0019] In some non-limiting embodiments or aspects, converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster may include: converting the at least one machine learning model from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library. The program instructions may also cause at least one processor to determine the new format based on a programming language used to operate the server cluster.

[0020] In some non-limiting embodiments or aspects, the initial performance metric generated at each of the at least two nodes may include false positive and false negative error rates based on the at least one converted machine learning model.

[0021] In some non-limiting embodiments or aspects, the at least one combined performance metric may include at least one of: AUROC; model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0022] In some non-limiting embodiments or aspects, the at least one model hyperparameter may include at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0023] Other non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0024] Clause 1: A computer-implemented method comprising: receiving, using at least one processor, a request identifying at least one machine learning model; initiating, using at least one processor, retrieval of the at least one machine learning model from a data repository based on the request; converting, using at least one processor, the at least one machine learning model from an initial format to an executable format compatible with distributed execution on a server cluster to generate at least one converted machine learning model; sending, using at least one processor, the at least one converted machine learning model to each of at least two nodes of the server cluster; executing, using at least one processor, the at least one converted machine learning model on each of the at least two nodes using data stored on the at least two nodes; executing, using at least one processor, the at least one converted machine learning model on each of the at least two nodes based on the data stored on the at least two nodes; The at least one converted machine learning model is executed on each of the nodes to generate an initial performance metric on each of the at least two nodes to produce a plurality of initial performance metrics; at least one processor is used to send the plurality of initial performance metrics from each of the at least two nodes to a processor outside the server cluster; at least one processor is used to combine the plurality of initial performance metrics to generate at least one combined performance metric of the at least one converted machine learning model; at least one processor is used to modify at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to generate at least one modified machine learning model; and at least one processor is used to execute the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0025] Clause 2: A computer-implemented method according to clause 1, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node further includes inputting the subset of data stored on each node into the at least one converted machine learning model based on the at least one data parameter.

[0026] Clause 3: A computer-implemented method according to Clause 1 or Clause 2, wherein converting the at least one machine learning model into the executable format compatible with distributed execution in the server cluster includes: utilizing at least one processor to convert the at least one machine learning model from the initial format based on a first machine learning programming library into a new format based on a second machine learning programming library different from the first machine learning programming library.

[0027] Clause 4: The computer-implemented method of any of clauses 1 to 3, further comprising determining, with at least one processor, the new format based on a programming language used to operate the server cluster.

[0028] Clause 5: A computer-implemented method according to any one of clauses 1 to 4, wherein the initial performance metric generated at each of the at least two nodes includes false positive and false negative error rates based on the at least one converted machine learning model.

[0029] Clause 6: A computer-implemented method according to any one of clauses 1 to 5, wherein the at least one combined performance metric comprises at least one of: area under receiver operating characteristic (AUROC); model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0030] Clause 7: A computer-implemented method according to any one of clauses 1 to 6, wherein the at least one model hyperparameter comprises at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0031] Item 8: A system comprising at least one server, the at least one server comprising at least one processor, wherein the at least one server is programmed or configured to: receive a request identifying at least one machine learning model; initiate retrieval of the at least one machine learning model from a data repository based on the request; convert the at least one machine learning model from an initial format to an executable format compatible with distributed execution on a server cluster to generate at least one converted machine learning model; send the at least one converted machine learning model to each of at least two nodes of the server cluster; execute the at least one converted machine learning model on each of the at least two nodes using data stored on each of the at least two nodes; generate an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to generate a plurality of initial performance metrics; send the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster; combine the plurality of initial performance metrics to generate at least one combined performance metric of the at least one converted machine learning model; modify at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to generate at least one modified machine learning model; and execute the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0032] Clause 9: A system according to clause 8, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node also includes inputting the subset of data stored on each node into the at least one converted machine learning model based on the at least one data parameter.

[0033] Clause 10: A system according to clause 8 or clause 9, wherein converting the at least one machine learning model into the executable format compatible with distributed execution in the server cluster includes: converting the at least one machine learning model from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library.

[0034] Clause 11: The system of any of clauses 8 to 10, wherein the at least one server is further programmed or configured to determine the new format based on a programming language used to operate the server cluster.

[0035] Clause 12: A system according to any one of clauses 8 to 11, wherein the initial performance metric generated at each of the at least two nodes includes false positive and false negative error rates based on the at least one converted machine learning model.

[0036] Clause 13: A system according to any one of clauses 8 to 12, wherein the at least one combined performance metric comprises at least one of: area under receiver operating characteristic (AUROC); model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0037] Clause 14: A system according to any one of clauses 8 to 13, wherein the at least one model hyperparameter comprises at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0038] Clause 15: A computer program product comprising at least one non-transitory computer-readable medium, the non-transitory computer-readable medium comprising program instructions, the program instructions, when executed by at least one processor, causes the at least one processor to: receive a request to identify at least one machine learning model; initiate retrieval of the at least one machine learning model from a data repository based on the request; convert the at least one machine learning model from an initial format to an executable format compatible with distributed execution on a server cluster to generate at least one converted machine learning model; send the at least one converted machine learning model to each of at least two nodes of the server cluster; execute the at least one converted machine learning model on each of the at least two nodes using data stored on the at least two nodes; at least one converted machine learning model; generating an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to produce a plurality of initial performance metrics; sending the plurality of initial performance metrics from each of the at least two nodes to a processor outside the server cluster; combining the plurality of initial performance metrics to produce at least one combined performance metric of the at least one converted machine learning model; modifying at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model; and executing the at least one modified machine learning model in a computer system to evaluate real-time event data.

[0039] Clause 16: A computer program product according to clause 15, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node also includes inputting the subset of data stored on each node to the at least one converted machine learning model based on the at least one data parameter.

[0040] Clause 17: A computer program product according to clause 15 or clause 16, wherein converting the at least one machine learning model into the executable format compatible with distributed execution in the server cluster includes: converting the at least one machine learning model from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library.

[0041] Clause 18: The computer program product of any of clauses 15 to 17, wherein the program instructions further cause the at least one processor to determine the new format based on a programming language used to operate the server cluster.

[0042] Clause 19: A computer program product according to any one of clauses 15 to 18, wherein the initial performance metric generated at each of the at least two nodes includes false positive and false negative error rates based on the at least one converted machine learning model.

[0043] Clause 20: A computer program product according to any one of clauses 15 to 19, wherein the at least one combined performance metric comprises at least one of: area under receiver operating characteristic (AUROC); model sensitivity; model specificity; false positive rate; false negative rate; error rate; F-score; or any combination thereof.

[0044] Clause 21: A computer program product according to any one of clauses 15 to 20, wherein the at least one model hyperparameter comprises at least one of: a classification threshold; a neural network topology; a neural network size; a learning rate; or any combination thereof.

[0045] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of combinations of related structural elements and parts and economies of manufacturing will become more apparent when considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals indicate corresponding parts in the various figures. However, it should be expressly understood that the drawings are for illustration and description purposes only and are not intended as a definition of the limitations of the present disclosure. Unless the context clearly dictates otherwise, when used in this specification and claims, the singular forms "a", "an", and "the" include plural referents. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Additional advantages and details of the present disclosure are explained in more detail below with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0047] Figure 1 is a schematic diagram of an overall environment for distributed execution of a machine learning model according to some non-limiting embodiments or aspects;

[0048] Figure 2 is a schematic diagram of a system for distributed execution of a machine learning model according to some non-limiting embodiments or aspects;

[0049] Figure 3 is a diagram of one or more components, devices, and / or systems according to some non-limiting embodiments or aspects;

[0050] Figure 4 is a flowchart of a method for distributed execution of a machine learning model according to some non-limiting embodiments or aspects; and

[0051] Figure 5is a flowchart of a method for distributed execution of a machine learning model according to some non-limiting embodiments or aspects.

[0052] Those skilled in the art will appreciate that any block diagrams herein represent conceptual views of illustrative systems that embody the principles of the disclosed subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocodes, etc. represent various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown. DETAILED DESCRIPTION

[0053] For the purposes of the following description, the terms "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the non-limiting embodiments or aspects as they are oriented in the accompanying drawings. However, it shall be understood that the non-limiting embodiments or aspects may employ various alternative variations and step sequences, except where expressly specified to the contrary. It shall also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary embodiments or aspects. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0054] Aspects, parts, elements, structures, actions, steps, functions, instructions and / or the like used herein should not be understood as critical or necessary unless explicitly described as such. Also, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of intending to have only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" and / or its like is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0055] Some non-limiting embodiments or aspects are described herein in conjunction with thresholds. As used herein, satisfying a threshold may refer to a value being greater than a threshold, more than a threshold, above a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, below a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0056] As used herein, the term "acquirer institution" may refer to an entity that is licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. Transactions that may be initiated by an acquirer institution may include payment transactions (e.g., purchases, original letter of credit transactions (OCTs), account funding transactions (AFTs), and / or the like). In some non-limiting embodiments or aspects, the acquirer institution may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer institution, such as a server computer executing one or more software applications.

[0057] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a replacement or substitute identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases and / or the like) such that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN may be associated with multiple tokens for different individuals or purposes.

[0058] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transmission, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, and / or the like) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit.

[0059] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include necessary components for receiving, processing, and outputting data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smart phone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer. An "application" or "application programming interface" (API) may refer to computer code or other data sorted on a computer-readable medium, which may be executed by a processor to facilitate interaction between software components, such as interaction of a client-side front end and / or a server-side back end for receiving data from a client. An "interface" may refer to a generated display, such as one or more graphical user interfaces (GUIs), with which a user may interact directly or indirectly (e.g., via a keyboard, a mouse, etc.).

[0060] As used herein, the terms "electronic wallet" and "electronic wallet application" refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device executing an electronic wallet application, and may also include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An "electronic wallet provider" may include an entity that provides and / or maintains electronic wallets for customers, such as Google Android Apple Samsung And / or other similar electronic payment systems. In some non-limiting examples, the issuing bank may be the electronic wallet provider.

[0061] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide customers with account identifiers, such as PANs, that uniquely identify one or more accounts associated with the customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0062] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The term "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications. As used herein, a "point of sale (POS) system" may refer to one or more computers and / or peripheral devices used by a merchant to conduct payment transactions with customers, including one or more card readers, scanning devices (e.g., code scanners), Communications receiver, near field communications (NFC) receiver, radio frequency identification (RFID) receiver and / or other contactless transceiver or receiver, contact-based receiver, payment terminal, computer, server, input device and / or other similar device that can be used to initiate a payment transaction.

[0063] As used herein, the term "payment device" may refer to a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or pendant, an RFID transponder, a retailer discount or membership card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, etc.).

[0064] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, wherein the entity (e.g., a merchant service provider, a payment service provider, a payment service provider, a payment service provider with a contract with an acquirer, a payment aggregator, and / or the like) provides payment services (e.g., a transaction service provider payment service, a payment processing service, and / or the like) to one or more merchants. The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, etc. operated by or on behalf of a payment gateway.

[0065] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that there may be various other arrangements. In addition, multiple computing devices (e.g., servers, POS devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system." As used herein, references to "servers" or "processors" may refer to previously described servers and / or processors, different servers and / or processors, and / or combinations of servers and / or processors that are stated to implement previous steps or functions. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different servers and / or processors stated to perform a second step or function.

[0066] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and an issuer organization. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server that executes one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0067] The systems, methods, and computer program products described herein provide many technical advantages and improvements in systems for executing machine learning models on multiple server nodes. As described herein, the present disclosure greatly reduces the computer resource requirements for executing a machine learning model from development, through training and testing, to in a production environment. The reduction in computer resource requirements includes, but is not limited to, bandwidth reduction, data retrieval time reduction, machine learning model execution time reduction, reduced changes in storage requirements at the processor executing the machine learning model, and the like. These benefits are provided by moving the machine learning model to stored data (e.g., retrieved from a repository and sent to a node for execution) rather than by moving the stored data to the machine learning model. The machine learning model is executed across multiple nodes of a server cluster that stores data for use as input to the machine learning model, which reduces runtime and allows for segmented calculation of performance metrics, thereby further providing the benefits.

[0068] The present disclosure also solves the technical problem of machine learning models developed in a variety of different formats (e.g., types of export files generated from a specific programming library), which may be incompatible with the format in which data is stored on a server cluster. In particular, the present disclosure converts a machine learning model from an initial format (e.g., a format exported or stored in a data repository) to an executable format compatible with distributed execution on a server cluster. The present disclosure improves the overall efficiency of the process flow involving machine learning model format conversion by converting each machine learning model in the steps between and / or including the retrieval of the machine learning model and the loading of the machine learning model in each node of the server cluster. In this way, the machine learning model can be developed and stored in a format that is most beneficial to the model developer, and the machine learning model can still be imported and loaded for execution in a format that can be executed in a distributed environment.

[0069] Reference now Figure 1 , shows an overall environment 100 for distributed execution of machine learning models according to some non-limiting embodiments or aspects. Specifically, the overall environment 100 may include a model distribution management system 201, a server cluster 212, and a user device 220 that communicate at least in part through one or more communication networks 203.

[0070] The model distribution management system 201 may include one or more computing devices for generating, training, executing, evaluating and / or implementing machine learning models. The model distribution management system 201 may include a model development system 202, a data repository 208, an orchestration system 210 and / or an implementation system 222. One or more systems depicted in the model distribution management system 201 may be included in the same system. The model distribution management system 201 may be configured to communicate with a user device 220 and / or a server cluster 212 at least in part through one or more communication networks 203. The model distribution management system 201 may include, be associated with, or be included in a transaction processing system. The described or depicted communications between the model distribution management system 201 and the server cluster 212 and / or the user device 220 may be sent to or from one or more systems included in the model distribution management system 201.

[0071] Model development system 202 may include one or more computing devices for initializing (e.g., writing code for a machine learning model) and / or initially training a machine learning model. Model development system 202 may be configured to communicate with data repository 208 and / or orchestration system 210 at least in part via one or more communication networks 203. A user (e.g., a data scientist) may use model development system 202 to generate a plurality of machine learning models to be tested. One or more machine learning models generated in model development system 202 may be exported and stored in data repository 208 in an initial format.

[0072] Data repository 208 may include one or more computing devices for storing machine learning models in an initial format. Data repository 208 may be configured to communicate with model development system 202, orchestration system 210, and / or implementation system 222. Data repository 208 may include one or more databases, data stores, etc. for storing data (e.g., of machine learning models) in a searchable arrangement. Machine learning models stored in data repository 208 may be retrieved based on one or more identifiers associated with the machine learning models. A user who wishes to evaluate a machine learning model stored in data repository 208 may use the one or more identifiers to identify the machine learning model in a request sent from user device 220. The machine learning model may be routinely retrieved from data repository 208, modified, and re-stored in data repository 208 by orchestration system 210.

[0073] Orchestration system 210 may include one or more computing devices for retrieving, sending, executing, evaluating, and / or modifying machine learning models in the depicted overall environment 100. Orchestration system 210 may be configured to communicate with model development system 202, data repository 208, server cluster 212 (including one or more server nodes 213a-213n thereof), user device 220, and / or implementation system 222. Orchestration system 210 may execute routines (e.g., processes, such as software algorithms) that bind other routines (e.g., model storage routines, model conversion routines, model execution routines, model evaluation routines, etc.) to create a workflow that is executed after a user configures all required components and triggers orchestration system 210. Orchestration system 210 may receive one or more requests from user device 220 to evaluate one or more machine learning models. Orchestration system 210 may collect model evaluation results from server cluster 212, generate a model report (e.g., performance metric results), and deliver the model report to user device 220.

[0074] The orchestration system 210 can execute an orchestration routine for each user request to evaluate a machine learning model. The orchestration routine can include multiple additional steps of receiving a machine learning model from the data repository 208 and facilitating distributed execution on the server cluster 212. First, the orchestration system 210 can infer the model type (e.g., format) of the requested machine learning model based on one or more parameters of the stored machine learning model. The orchestration system 210 can obtain the model binary file from the model storage area (e.g., data repository 208) and then perform the conversion routine using the model type as input. The orchestration system 210 can then read the user configuration in the request based on the user-defined data elements. Based on the user request, the orchestration system 210 can also identify the data partitions (e.g., nodes) of the server cluster 212, which contain data that will be used as input for evaluating, training, testing, etc. the machine learning model.

[0075] Orchestration system 210 can cause execution of an execution routine on one or more nodes of server cluster 212 based on each user request to evaluate a machine learning model. An execution routine can be a function that accepts a data row (e.g., a record stored on a server node) and a model as input. If the machine learning model is not initialized, orchestration system 210 can execute a container initialization function, load the model binary, and trigger the model binary. Orchestration system 210 can then run different model execution routines based on the original model type (e.g., format). For example, if the original model is If the model is exported, orchestration system 210 can convert the data row input into a tensor, which is a multidimensional array with a uniform type. Orchestration system 210 can then cause the server node to run the model execution function using the tensor as input. As another example, if the original model type is H2O TM The orchestration system 210 can convert the data row input into H2O TM Row data class, and have the server node run the model execution function using the row data class object as input. As another example, if the original model type is Exported model or Spark TM If a PMML-derived model is used, orchestration system 210 can convert the data row input into a PMML field mapping and enable the server node to execute the model execution function using the mapping object as input.

[0076] Orchestration system 210 may cause the evaluation routine to be executed on one or more nodes of server cluster 212 after the execution routine. For example, in the evaluation routine, the server node may obtain the results from the execution routine and load the results into Spark TMDataFrame, reads any received user configuration, reads the model configuration, fetches additional datasets (if required for performance reporting) and loads those datasets into Spark TM Data frame, and execute the evaluation function. The evaluation function may include one or more initial (e.g., partial) metric functions, including but not limited to score distribution metric function, accuracy metric function, sensitivity metric function, specificity metric function, F score metric function, AUROC metric function, etc. Then, the computing device may build a model architecture, which may include a model identifier, model configuration, hyperparameters, and initial performance metric objects. The computing device may also convert the execution results from the evaluation logic into an architecture format and push the data to a storage device (e.g., in a server node).

[0077] The orchestration system 210 may perform a feature impact analysis routine after the execution and evaluation routine. The feature impact analysis routine may be performed by the orchestration system 210. Arrangement importance logic may be used for the feature impact analysis routine. In this routine, for each machine learning model and each feature of the machine learning model (e.g., an independent variable that serves as an input to the machine learning model), the selection of input data may be disrupted and re-entered using the execution and evaluation routine described above. Using the old and new outputs of the machine learning model, the impact of the input change may be calculated using a loss function, and the weight of each model feature may be determined. After calculating the weight of each model feature of each model, the results may be aggregated to find the overall impact of each model feature. The results may be output (e.g., to the user device 220, the model development system 202, etc.) for use by the user (e.g., a data scientist) in a subsequent training process and / or identifying the best model features.

[0078] The orchestration system 210 may perform a hyperparameter tuning routine after the execution and evaluation routine. When training multiple models, a large amount of data may make it time-consuming or resource-consuming to identify the best initial hyperparameters. Therefore, a model may be initially created using random hyperparameters (e.g., in the model development system 202), and then the hyperparameter tuning routine may help identify the best hyperparameters. For example, for each model and for each hyperparameter value, the orchestration system 210 may assign a score to the hyperparameter value. The score may be calculated using a combination of a combined performance metric and a combination of hyperparameter values. In order to calculate the hyperparameter score associated with each pre-trained value, the orchestration system 210 may use a Gaussian process as an alternative to learn the mapping from the hyperparameter configuration to the performance metric. The orchestration system 210 may follow the first step of Gaussian process optimization of a single hyperparameter variable such as a learning rate, batch size, number of layers, and drop rate. The Gaussian process may be an algorithm based on the Sequential Model Based Optimization (SMBO) class. After determining the hyperparameter value scores and rankings, the value rankings may be stored and / or provided to a user (eg, via the user device 220 , the model development system 202 , etc.).

[0079] The orchestration system 210 may perform an integrated analysis routine after the execution and evaluation routines of multiple machine learning models. For example, a user may train multiple machine learning models and want to find out which machine learning model is used in a production environment (e.g., implementation system 222). The evaluation routine described above may be used for each machine learning model to determine individual model performance. The integrated analysis routine may allow for the combination of different models and allow for the generation of scores for the combination of different models in order to produce the best model output (e.g., prediction). For example, a Bayesian approach may be used to find the best combination of machine learning models. The Bayesian approach works by finding the next set of weights to evaluate the actual objective function by selecting the weights that perform best for the surrogate function. This Bayesian approach may determine the best integrated combination of machine learning models, which may work better than individual machine learning models. The integrated analysis routine may include retrieving all model performance metrics for the same input data set, obtaining a weight range for the model input (e.g., defined by the user), and generating a grid with all possible combinations of models and weights. The integrated analysis routine may also include, for each combination of model and weight: determining the average score of the grid row; running performance analysis; and saving performance analysis. The performance results of the integrated analysis may be provided to a user (eg, via user device 220, model development system 202, etc.).

[0080] The implementation system 222 may include one or more computing devices for executing at least one machine learning model to evaluate real-time event data. The implementation system 222 may be associated with a production environment (e.g., compared to a test environment) to implement services, such as services for a transaction service provider or associated with a transaction service provider. For example, the implementation system 222 may be a fraud detection system, and the executed machine learning model may be used to detect fraud in real-time transaction data. As another example, the implementation system 222 may be a credit extension system, and the executed machine learning model may be used to determine whether credit should be extended for a credit type transaction based on real-time transaction data. In another example, the implementation system 222 may be a computer-driven advertising system, and the executed machine learning model may be used to trigger relevant advertisements to payment device users based on real-time transaction data. It should be understood that many arrangements and configurations are possible. The implementation system 222 may be configured to communicate with the data repository 208 to receive a machine learning model from the data repository 208 for execution in a production environment. The implementation system 222 may be configured to communicate with the orchestration system 210 to receive a modified machine learning model for execution in a production environment. The implementation system 222 may also be configured to communicate with the user device 220 to configure the machine learning model for execution on the real-time event data.

[0081] The server cluster 212 may include a plurality of server nodes 213a-213n, on which the machine learning model may be distributed for execution. The server cluster 212 may store event data (e.g., transaction data). The server cluster 212 may be configured to communicate with the orchestration system 210 so that the converted machine learning model may be loaded, executed, and evaluated on each of the plurality of nodes 213a-213n. Each of the plurality of nodes 213a-213n may store a subset of data that may be used to train and / or evaluate the machine learning model. For example, the first node 213a of the server cluster 212 may store a first subset of data, and the machine learning model may be retrieved by the orchestration system 210, converted by the orchestration system 210, and imported into the first node 213a in a first loading routine 214a. Then, the first node 213a may execute a first execution routine 216a to input the data subset from the first node 213a to the machine learning model. The first node 213a may then execute a first evaluation routine 218a to determine an initial performance metric for the machine learning model executed on the first subset of data stored on the first node 213a. The plurality of nodes 213a-213n may have corresponding loading routines 214-214n, execution routines 216a-216n, and evaluation routines 218a-218n. When evaluating the machine learning model, the orchestration system 210 may cause the machine learning model to be converted, loaded, executed, and evaluated across one or more of the plurality of server nodes 213a-213n, which may be all of the server nodes 213a-213n or less than the total number of the server nodes.

[0082] If the type of machine learning model requires, orchestration system 210 can communicate with server cluster 212 to repartition the data stored on the server cluster to optimize the execution of the machine learning model. For example, if the machine learning model is a convolutional neural network (CNN), a perceptron model, or an autoencoder model, orchestration system 210 can read the model configuration, infer the data architecture, execute functions to determine the difference in architecture, and if a difference in architecture is detected, orchestration system 210 can repartition the data on server cluster 212. If no difference in architecture is detected, orchestration system 210 may not repartition the data. As another example, if the machine learning model is a recurrent neural network (RNN), orchestration system 210 can read the model configuration, infer the data schema, execute a sequencer function, and repartition the data.

[0083] If repartitioning is required, orchestration system 210 can trigger a repartitioning function, obtain feature engineering configuration, build feature engineering logic, build execution packages for repartitioning, and assign execution packages to existing partitions (e.g., nodes) for repartitioning data thereon.

[0084] When orchestration system 210 has determined the partitions on which to execute the machine learning model, for each of these partitions, orchestration system 210 can perform the user-defined data transformations requested in the user request. Such transformation routines can include obtaining the original model type and the required model type, checking the compatibility of the original model type and the required model type, and converting the original model type to the required model type when applicable. After the transformation routine, orchestration system 210 can execute the feature engineering function for each partition, execute the model execution routine, and execute the model evaluation routine.

[0085] User device 220 may include one or more computing devices configured to communicate with orchestration system 210 at least in part over one or more communication networks 203. User device 220 may include a user interface (e.g., one or more applications, web pages, etc.) to allow a user to interact in overall environment 100. User device 220 may send one or more requests to evaluate a machine learning model (e.g., to orchestration system 210), and such request may include one or more identifiers to identify one or more machine learning models to be evaluated. The request may include or be included in a message, data packet, data signal, etc. sent from user device 220 to orchestration system 210. User device 220 may include a display for displaying a report of the evaluated machine learning model. User device 220 may also be configured with a display to receive input from a user to modify one or more parameters of the machine learning model being evaluated, receive input to further specify the scope of evaluating the machine learning model, and display the results (e.g., report) of the evaluated machine learning model. User device 220 may also be configured to communicate with implementation system 222 at least in part over one or more communication networks 203 to set up and execute the trained machine learning model in a production environment.

[0086] The depicted environment 100 may be implemented with the aforementioned components communicating across one or more communication networks 203. The one or more communication networks 203 may include one or more wired and / or wireless networks. For example, the communication network 203 may include a cellular network (e.g., Long Term Evolution). networks, third generation (3G) networks, fourth generation (4G) networks, fifth generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks, ad hoc networks, mesh networks, beacon networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0087] Reference now Figure 2 , shows a system 200 for distributed execution of machine learning models according to some non-limiting embodiments or aspects. Specifically, the system 200 may include a model development system 202, a data repository 208, an orchestration system 210, a server cluster 212 having a plurality of server nodes 213a-213n, a user device 220, and an implementation system 222.

[0088] The model development system 202 may also include a plurality of machine learning development frameworks 204a-204n (also referred to herein as machine learning programming libraries, such as H2O TM , Keras TM 、Spark TM ML, etc.). The plurality of machine learning development frameworks 204a-204n may be configured to generate a corresponding plurality of formats 206a-206n (e.g., model types) of the exported machine learning models (e.g., Generate SavedModel format, The one or more machine learning models generated in the model development system 202 may be exported in the initial format 206a-206n and stored in the data repository 208.

[0089] Orchestration system 210 provides technical benefits that are independent of model format, because the processes performed by orchestration system 210 allow testing that is independent of model type or deep learning architecture type. For example, orchestration system 210 can execute a conversion routine that converts a machine learning model from an initial format stored in data repository 208 to a second format that is compatible with execution in a distributed environment (e.g., in server cluster 212). As another example, if server cluster 212 is configured to run in a Java virtual machine , the model can be converted to Custom conversion functions can be written in Spark TMAbove, in order to convert from the format of one or more of the plurality of machine learning development frameworks 204a-204n. The orchestration system 210 can also construct dynamic containers that perform other processes (eg, routines) described herein.

[0090] The conversion routine may be a function of the orchestration system 210 that accepts a model type (e.g., format) as input. Based on the model type, the conversion routine may perform different steps to convert from the original model type to the target model type. For example, if the original model type is The exported model can then be loaded by the orchestration system 210 Serve binary files, convert binary files to executable file, build an executable container, and The exported model is dynamically embedded in the container for execution on the server node. As another example, if the original model type is H2O TM The exported model, the orchestration system 210 can obtain H2O TM , infer the binary file, convert the binary file to executable file, build an executable container, and install H2O TM The exported model is dynamically embedded in the container for execution on the server node. As another example, if the original model type is The exported model, the orchestration system 210 can load the Predictive Model Markup Language (PMML) binary file and convert the binary file into executable file, build an executable container, and The exported model is dynamically embedded in the container for execution on the server node. As another example, if the original model type is Spark TM ML export model, the orchestration system 210 can obtain PMML binary file, convert binary file to Executable files, build executable containers, and Spark TM The ML exported model is dynamically embedded in the container for execution on the server node.

[0091] A method for distributed execution of machine learning models in server cluster 212 may be performed in depicted system 200. Orchestration system 210 may receive a request from user device 220 to identify at least one machine learning model. The request may further identify data in server cluster 212 (e.g., based on a category or parameter of the data, such as retail transaction data, grocery transaction data, etc.) to input into the machine learning model for evaluating the machine learning model. In response to receiving the request, orchestration system 210 may retrieve or initiate retrieval of at least one machine learning model from data repository 208 based on the request. Before, during, or after retrieving at least one machine learning model from data repository 208, orchestration system 210 may convert at least one machine learning model from an initial format (e.g., stored in data repository 208) to an executable format compatible with distributed execution on server cluster 212. Orchestration system 210 may thereby generate at least one converted machine learning model.

[0092] After converting at least one machine learning model, orchestration system 210 can determine one or more nodes in a plurality of server nodes 213a-213n of server cluster 212 that contain data to be used as input and to evaluate the machine learning model. If the user specifies the data to be used to evaluate the model (e.g., in the request), orchestration system 210 can refer to the specification and identify where the data is stored among the partitioned data stored in server cluster 212, and identify the corresponding node (e.g., on which the data is stored) to receive the machine learning model. Based on the node identified by orchestration system 210 to receive the model, orchestration system 210 can send at least one converted machine learning model to one or more nodes in a plurality of server nodes 213a-213n.

[0093] After one or more nodes of the server cluster 212 receive the converted machine learning model, the orchestration system 210 may execute or cause execution of at least one converted machine learning model on each node that receives the converted machine learning model using the data stored on the corresponding node. For example, if it is determined that the first node 213a has data used as input for evaluating the machine learning model, the first node 213a may receive the converted machine learning model and execute the converted machine learning model on the first node 213a using the data stored on the first node 213a as input. The data used as input to the converted machine learning model on a given node may be all or a subset of the data stored on the given node. Based on the execution of the converted machine learning model, the orchestration system 210 may generate or cause the generation of initial performance metrics on each node that receives and executes the converted machine learning model. The initial performance metrics may be based on the execution of the converted machine learning model on the corresponding node of the server cluster 212.

[0094] For example, an individual initial performance metric can be generated on each node executing the converted machine learning model. The metric can be generated in the same container as the model being evaluated. The initial performance metric on a given node may also be referred to herein as a partial performance metric because the initial performance metrics can be combined across nodes to generate a combined performance metric that reflects the evaluation of the machine learning model executed across all relevant nodes. Since the data used to evaluate the model and generate the initial performance metric is only a small portion of all the data stored in the server cluster 212, network traffic can be significantly reduced.

[0095] As another example, the machine learning model evaluated can produce an output of a binary classifier type. A binary classifier can predict all data examples of a test data set as positive or negative. Classification (e.g., prediction) can produce four results: true positive (e.g., correct positive prediction), true negative (e.g., correct negative prediction), false positive (e.g., incorrect positive prediction) and false negative (e.g., incorrect negative prediction). The evaluation routine on the server node can perform the following process to generate an initial performance metric: (i) streaming labels from a source data set to an evaluation container (e.g., the node); (ii) calculating the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN) of available data in the container (e.g., the node); and (iii) calculating the initial performance metric based on TP, TN, FP, and FN. For illustration, an error rate can be used as an initial performance metric, which can be calculated according to the following formula:

[0096] Formula 1

[0097]

[0098] The aforementioned evaluation routine may be performed on each node that receives and executes the converted machine learning model. Each node may send the results of such evaluation to the orchestration system 210. For example, a data object having the following data structure may be created at each node to convey the results of the evaluation: {data partition identifier, category, compute_metric, list <supporting data points>}, where "data partition identifier" represents the node, "category" represents the subset of data used as input to the machine learning model on the node, "compute_metric" represents one or more initial performance metrics, and "list <supporting data points>" represents one or more data portions included to verify the generated initial performance metrics, which may include FP, FN, TP, or TN. The data object having the aforementioned data structure may be sent to the orchestration system 210 from each node used to evaluate the machine learning model.

[0099] After orchestration system 210 receives multiple initial performance metrics, orchestration system 210 can combine the multiple initial performance metrics to generate at least one combined performance metric of the converted machine learning model, which can represent the performance of the machine learning model on all data input to the machine learning model across all nodes. Orchestration system 210 can wait until all evaluation routines are completed to generate the combined performance metric. The combined performance metric can use the same formula as the initial performance metric, but for the entire data set (for example, by summing the TP, TN, FP, and FN of all initial performance metrics and using Formula 1 to calculate the error rate for the entire data set). Multiple initial performance metrics and / or combined performance metrics can be sent from orchestration system 210 to user device 220 for display to the user, thereby providing evaluation feedback. Combined performance metrics may include, but are not limited to: AUROC; model sensitivity (e.g., model recall, which is the true positive rate, which is the number of true positives divided by the sum of true positives and false negatives); model specificity (e.g., true negative rate, which is the number of true negatives divided by the sum of true negatives and false positives); false positive rate (e.g., the number of false positives divided by the sum of true negatives and false positives); false negative rate (e.g., the number of false negatives divided by the sum of true positives and false negatives); error rate; F score (e.g., F1 score, F2 score, F3 score); β score, etc.); model accuracy (e.g., the number of true positives divided by the sum of true positives and false negatives); or any combination thereof. The F score can be calculated according to the following equation:

[0100] Formula 2

[0101]

[0102] Where "PREC" represents model precision (e.g., the number of true positives divided by the sum of true positives and false negatives), "REC" represents model recall (e.g., the number of true positives divided by the sum of true positives and false negatives), and β is a positive real factor (e.g., a weight) selected so that recall is treated as β times precision. If multiple combined performance metrics are generated, the multiple performance metrics can be presented to the user.

[0103] Based on at least one combined performance metric, orchestration system 210 can modify at least one model hyperparameter of at least one machine learning model to produce a modified machine learning model. Hyperparameters can include, but are not limited to: classification thresholds (e.g., decision thresholds used to convert quantitative model outputs into classifier outputs); neural network topology (e.g., adding, removing, or modifying one or more connections of a node of a neural network); neural network size (e.g., adding, removing, or modifying a node of a neural network); learning rate (e.g., the amount by which the model is changed in response to an estimated error each time a model weight is updated); or any combination thereof. For example, a higher error rate may result in more modifications to the neural network topology or neural network size, more significant changes in thresholds, and / or more significant shifts in learning rates to address the higher error rate.

[0104] After modifying the machine learning model, the modified machine learning model can be re-stored in the data repository 208 and / or re-assigned to the server cluster 212 for additional training and testing. It should be appreciated that the machine learning model can be placed in a test and training loop (e.g., a repeating loop) such that the machine learning model is repeatedly sent to one or more nodes of the server cluster 212, executed on one or more nodes of the server cluster 212, evaluated on one or more nodes of the server cluster 212, evaluated at the orchestration system 210 using one or more combined performance metrics, and modified based on the one or more combined performance metrics. For example, at the beginning of each loop, after modifying the machine learning model, the orchestration system 210 can send the machine learning model to the same or different subset of nodes in the plurality of nodes 213a-213n of the server cluster 212. The orchestration system 210 can then cause the subset of nodes to execute the modified machine learning model using the data stored on the subset of nodes as input to the modified machine learning model. The subset of nodes can then evaluate the modified machine learning model by generating an initial performance metric for each node in the subset of nodes based on executing the modified machine learning model on the node. Orchestration system 210 may receive multiple initial performance metrics from the node subset and may determine a combined performance metric based on the multiple initial performance metrics. Based on the combined performance metric, the modified machine learning model may be further modified and orchestration system 210 may start the cycle again.

[0105] The testing and training loop can provide the additional benefit of iterative improvement to optimize the performance of the machine learning model. After one or more cycles of the above method, the model can be determined to be sufficiently accurate (e.g., sufficiently low error rate). After modification, the machine learning model can be executed in the implementation system 222 (e.g., a production environment) to evaluate real-time event data.

[0106] Reference now Figure 3, a diagram showing example components of an apparatus 300 according to some non-limiting embodiments or aspects. The apparatus 300 may correspond to one or more of the following: a model distribution management system 201, a model development system 202, a data repository 208, an orchestration system 210, a server cluster 212, a user device 220, an implementation system 222, and / or one or more communication networks 203 in which the environment 100 and / or the system 200 operate, such as Figure 1 and 2 In some non-limiting embodiments or aspects, such a system or device may include at least one device 300 and / or at least one component of a device 300. Figure 3 In some non-limiting embodiments or aspects, the number and arrangement of components shown in Figure 3 Compared to those shown in , the device 300 may include additional components, fewer components, different components, or components arranged in a different manner. Additionally or alternatively, a group of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another group of components of the device 300.

[0107] like Figure 3 As shown, the device 300 may include a bus 302, a processor 304, a memory 306, a storage component 308, an input component 310, an output component 312, and a communication interface 314. The bus 302 may include components that permit communication between components of the device 300. In some non-limiting embodiments or aspects, the processor 304 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 304 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 306 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 304.

[0108] Continue to refer Figure 3, the storage component 308 may store information and / or software related to the operation and use of the device 300. For example, the storage component 308 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 310 may include a component that permits the device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 310 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 312 may include a component that provides output information from the device 300 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 314 may include a transceiver-type component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables the device 300 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 314 may permit the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 314 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, Interface, cellular network interface, etc.

[0109] Device 300 can perform one or more processes described herein. Device 300 can perform these processes based on processor 304 executing software instructions stored by computer-readable media such as memory 306 and / or storage component 308. Computer-readable media can include any non-transient memory device. The memory device includes a memory space located in a single physical storage device or a memory space extended across multiple physical storage devices. Software instructions can be read from another computer-readable medium or from another device to memory 306 and / or storage component 308 via communication interface 314. When executed, the software instructions stored in memory 306 and / or storage component 308 can enable processor 304 to perform one or more processes described herein. In addition or alternatively, hard-wired circuit systems can be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments or aspects described herein are not limited to any specific combination of hardware circuits and software. As used herein, the term "programmed or configured" refers to the arrangement of software, hardware circuits, or any combination thereof on one or more devices.

[0110] Reference now Figure 4, a flowchart of a method 400 for distributed execution of a machine learning model on a server cluster according to some non-limiting embodiments or aspects of the present disclosure is shown. The method 400 may be performed by a model distribution management system 201, a model development system 202, a data repository 208, an orchestration system 210, a server cluster 212 (including one or more nodes 213a-213n thereof), a user device 220, and / or another computing device. One or more steps performed by the first processor may be performed by the same or different processors. Figure 4 The steps shown are for example purposes only. It should be understood that additional, fewer, different and / or different order steps may be used in non-limiting embodiments or aspects.

[0111] In step 402, a request identifying at least one machine learning model may be received. For example, orchestration system 210 may receive a request identifying at least one machine learning model from user device 220. The request may include a message including one or more model identifiers and / or at least one data parameter associated with a subset of data stored in server cluster 212. The one or more model identifiers in the user request may be used to identify one or more machine learning models stored in data repository 208, so that the one or more machine learning models may be retrieved for execution, evaluation, modification, etc. At least one data parameter associated with a subset of data stored in server cluster 212 may be used to identify which nodes of server cluster 212 execute the one or more machine learning models. The user request may also be multiple user requests.

[0112] In step 404, retrieval of the machine learning model may be initiated. For example, in response to receiving a user request from user device 220, orchestration system 210 may initiate retrieval of at least one machine learning model from data repository 208 based on the user request. As another example, orchestration system 210 may retrieve one or more machine learning models from data repository 208 based on one or more model identifiers corresponding to the one or more machine learning models included in the user request. The one or more machine learning models stored in data repository 208 may be stored in an initial format before orchestration system 210 retrieves them.

[0113] In step 406, the new format may be determined based on the programming language used to operate server cluster 212. For example, orchestration system 210 may determine the programming language used to operate server cluster 212 (e.g., Scala TM , Python TMThen, the orchestration system 210 can determine a new format based on the programming language used to operate the server cluster 212, and the machine learning model is converted to the new format. The new format can include data objects and libraries that are compatible with the input and used in the programming language of the server cluster 212.

[0114] In step 408, the machine learning model can be converted from the initial format to a new format. For example, orchestration system 210 can convert at least one machine learning model from the initial format to an executable format compatible with distributed execution on server cluster 212 to produce at least one converted machine learning model. In some non-limiting embodiments or aspects, the new format can be determined by orchestration system 210 in step 406, the user request can specify the new format, server cluster 212 can identify the new format in a communication with orchestration system 210, etc. The initial format can be based on a first machine learning programming library (e.g., H2O TM , Keras TM 、Spark TM ML, etc.) (e.g., compatible with a first machine learning programming library, compiled using the first machine learning programming library, etc.), and the executable format may be a new format based on a second machine learning programming library that is different from the first machine learning programming library. The new format based on the second machine learning programming library (e.g., compatible with the second machine learning programming library, compiled using the second machine learning programming library, etc.) may be based on a programming language used to operate a server cluster.

[0115] In step 410, the converted machine learning model can be sent to one or more nodes of server cluster 212. For example, after conversion of the machine learning model, orchestration system 210 can send at least one converted machine learning model to each of at least two nodes (e.g., first node 213a, second node 213b, or more) of server cluster 212. Orchestration system 210 can build a container to execute the converted machine learning model on each node that receives the converted machine learning model. Orchestration system 210 can then cause each node that receives the converted machine learning model to execute the machine learning model on a subset of data stored on the node.

[0116] In step 412, the converted machine learning model can be executed on one or more nodes of server cluster 212 that received the converted machine learning model. For example, each node that receives the converted machine learning model in step 4120 can then execute at least one converted machine learning model on the node using the data stored on each node. All or some of the data stored on each node that executes the converted machine learning model can be input to the machine learning model. The data used as input to the converted machine learning model can be specified by orchestration system 210. The user request from user device 220 to orchestration system 210 in step 402 can instruct orchestration system 210 to specify the data used as input to the converted machine learning model.

[0117] After executing the converted machine learning model on each node that receives the converted machine learning model, method 400 can proceed to combining Figure 5 Additional steps of method 500 are depicted.

[0118] Reference now Figure 5 , a flowchart of a method 500 for distributed execution of a machine learning model on a server cluster according to some non-limiting embodiments or aspects of the present disclosure is shown. The method 500 can be performed by the model distribution management system 201, the model development system 202, the data repository 208, the orchestration system 210, the server cluster 212 (including one or more nodes 213a-213n thereof), the user device 220, and / or another computing device. One or more steps performed by the first processor can be performed by the same or different processors. Figure 5 The steps shown in are for illustrative purposes only. It should be understood that in non-limiting embodiments or aspects, additional, fewer, different and / or different orders of steps may be used. Method 500 may be interpreted as combining Figure 4 A continuation of method 400 is depicted.

[0119] In step 502, initial performance metrics can be generated on each node. For example, each node that executes the converted machine learning model can generate one or more initial performance metrics on the node based on executing at least one converted machine learning model on the node. In doing so, one or more nodes of the server cluster 212 can produce multiple initial performance metrics. At least one initial performance metric generated on each of the at least two nodes may include false positive and false negative error rates based on at least one converted machine learning model. The multiple initial performance metrics generated on one or more nodes of the server cluster 212 can be sent and / or retrieved from the nodes to the orchestration system 210.

[0120] In step 504, the plurality of initial performance metrics may be sent to a processor external to the server cluster. For example, each of the at least two nodes of server cluster 212 may send its respective initial performance metric to a processor external to the server cluster, such as orchestration system 210. Additionally or alternatively, orchestration system 210 may send the plurality of initial performance metrics received from the at least two nodes of server cluster 212 to a different processor external to the server cluster. The plurality of initial performance metrics may then be used to determine a combined (e.g., final) performance metric.

[0121] In step 506, multiple initial performance metrics can be combined. For example, orchestration system 210 can combine multiple initial performance metrics received from server cluster 212 to generate at least one combined performance metric of at least one converted machine learning model. The at least one combined performance metric can include, but is not limited to, AUROC, model sensitivity, model specificity, false positive rate, false negative rate, error rate, F score, or any combination thereof. The combined performance metric can indicate the overall performance of the evaluated machine learning model, through which parameters (e.g., hyperparameters) of the machine learning model can be adjusted to further improve the model.

[0122] In step 508, at least one model hyperparameter of the machine learning model can be modified based on the at least one combined performance metric. For example, orchestration system 210 can modify at least one model hyperparameter of at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model. The at least one model hyperparameter can include, but is not limited to, a classification threshold, a neural network topology, a neural network size, a learning rate, or any combination thereof. The hyperparameter can be associated with the performance metric such that adjustments to the hyperparameter are intended to further improve the performance metric when the machine learning model is executed again (e.g., for further evaluation or in a production environment).

[0123] In step 510, at least one modified machine learning model may be executed. For example, orchestration system 210 may execute at least one machine learning model in a computer system (e.g., implementation system 222) to evaluate real-time event data. It should be appreciated that execution may include initiating or causing execution of at least one modified machine learning model. The computer system may include or be included in system 200 or environment 100. Alternatively, the computer system may be separate from system 200 and environment 100. Real-time event data may include, but is not limited to, transaction data received and processed in real time by a transaction processing system.

[0124] Although the present disclosure has been described in detail based on what are currently considered to be the most practical and preferred embodiments or aspects for the purpose of illustration, it should be understood that such details are for that purpose only, and the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect, and that one or more steps may be performed in an order different from the order presented in the present disclosure.

Claims

1. A computer-implemented method comprising: receiving, with at least one processor, a request identifying at least one machine learning model; initiating, with at least one processor, retrieval of the at least one machine learning model from a data repository based on the request; converting, using at least one processor, the at least one machine learning model from an initial format to an executable format compatible with distributed execution on the server cluster to produce at least one converted machine learning model; sending, using at least one processor, the at least one converted machine learning model to each of at least two nodes of the server cluster; executing, with at least one processor, the at least one converted machine learning model on each of the at least two nodes using data stored on the each node; generating, with at least one processor, an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to produce a plurality of initial performance metrics; sending, using at least one processor, the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster; combining, with at least one processor, the plurality of initial performance metrics to produce at least one combined performance metric of the at least one converted machine learning model; modifying, with at least one processor, at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model; as well as The at least one modified machine learning model is executed in a computer system using at least one processor to evaluate real-time event data.

2. A computer-implemented method according to claim 1, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node also includes inputting the data subset stored on each node into the at least one converted machine learning model based on the at least one data parameter.

3. The computer-implemented method of claim 1 , wherein converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster comprises: The at least one machine learning model is converted, using at least one processor, from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library. 4 . The computer-implemented method of claim 3 , further comprising determining, with at least one processor, the new format based on a programming language used to operate the server cluster.

5. A computer-implemented method according to claim 1, wherein the initial performance metric generated at each of the at least two nodes includes false positive and false negative error rates based on the at least one converted machine learning model.

6. The computer-implemented method of claim 1 , wherein the at least one combined performance metric comprises at least one of: Area under the receiver operating characteristic (AUROC); Model sensitivity; Model specificity; False positive rate; False negative rate; Error rate; F score; or Any combination thereof.

7. The computer-implemented method of claim 6, wherein the at least one model hyperparameter comprises at least one of: Classification threshold; Neural network topology; Neural network size; learning rate; or Any combination thereof.

8. A system comprising at least one server, the at least one server comprising at least one processor, wherein the at least one server is programmed or configured to: receiving a request identifying at least one machine learning model; initiating retrieval of the at least one machine learning model from a data repository based on the request; converting the at least one machine learning model from an initial format into an executable format compatible with distributed execution on the server cluster to produce at least one converted machine learning model; sending the at least one converted machine learning model to each of at least two nodes of the server cluster; executing the at least one converted machine learning model on each of the at least two nodes using data stored on the each node; generating an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to produce a plurality of initial performance metrics; sending the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster; combining the plurality of initial performance metrics to produce at least one combined performance metric of the at least one converted machine learning model; modifying at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model; as well as The at least one modified machine learning model is executed in a computer system to evaluate real-time event data.

9. A system according to claim 8, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node also includes inputting the subset of data stored on each node into the at least one converted machine learning model based on the at least one data parameter.

10. The system of claim 8, wherein converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster comprises: The at least one machine learning model is converted from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library.

11. The system of claim 10, wherein the at least one server is further programmed or configured to determine the new format based on a programming language used to operate the server cluster.

12. A system according to claim 8, wherein the initial performance metric generated at each of the at least two nodes includes false positive and false negative error rates based on the at least one converted machine learning model.

13. The system of claim 8, wherein the at least one combined performance metric comprises at least one of: Area under the receiver operating characteristic (AUROC); Model sensitivity; Model specificity; False positive rate; False negative rate; Error rate; F score; or Any combination thereof.

14. The system of claim 13, wherein the at least one model hyperparameter comprises at least one of the following: Classification threshold; Neural network topology; Neural network size; learning rate; or Any combination thereof.

15. A computer program product comprising at least one non-transitory computer readable medium, the non-transitory computer readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: receiving a request identifying at least one machine learning model; initiating retrieval of the at least one machine learning model from a data repository based on the request; converting the at least one machine learning model from an initial format into an executable format compatible with distributed execution on the server cluster to produce at least one converted machine learning model; sending the at least one converted machine learning model to each of at least two nodes of the server cluster; executing the at least one converted machine learning model on each of the at least two nodes using data stored on the each node; generating an initial performance metric on each of the at least two nodes based on executing the at least one converted machine learning model on each of the at least two nodes to produce a plurality of initial performance metrics; sending the plurality of initial performance metrics from each of the at least two nodes to a processor external to the server cluster; combining the plurality of initial performance metrics to produce at least one combined performance metric of the at least one converted machine learning model; modifying at least one model hyperparameter of the at least one machine learning model based on the at least one combined performance metric to produce at least one modified machine learning model; as well as The at least one modified machine learning model is executed in a computer system to evaluate real-time event data.

16. A computer program product according to claim 15, wherein the request also includes at least one data parameter associated with a subset of data stored in the server cluster, and wherein executing the at least one converted machine learning model on each of the at least one node also includes inputting the subset of data stored on each node into the at least one converted machine learning model based on the at least one data parameter.

17. The computer program product of claim 15, wherein converting the at least one machine learning model to the executable format compatible with distributed execution in the server cluster comprises: The at least one machine learning model is converted from the initial format based on a first machine learning programming library to a new format based on a second machine learning programming library different from the first machine learning programming library.

18. The computer program product of claim 17, wherein the program instructions further cause the at least one processor to determine the new format based on a programming language used to operate the server cluster.

19. The computer program product of claim 15, wherein the at least one combined performance metric comprises at least one of: Area under the receiver operating characteristic (AUROC); Model sensitivity; Model specificity; False positive rate; False negative rate; Error rate; F score; or Any combination thereof.

20. The computer program product of claim 19, wherein the at least one model hyperparameter comprises at least one of: Classification threshold; Neural network topology; Neural network size; learning rate; or Any combination thereof.

Images