Implementing model agnostic framework to provide Shapril values associated with machine learning model

Abstract

Methods, systems, and computer program products are provided for implementing a model agnostic framework to provide Shapril values associated with a machine learning model. A method may include receiving an executable file for a neural network machine learning model; converting the format of the executable file used for the neural network machine learning model into an agnostic model format so as to provide an agnostic model format file for the neural network machine learning model; parsing the model format agnostic file to provide a forward symbol graph associated with the neural network machine learning model and a reverse symbol graph associated with the neural network machine learning model; receiving a real-time inference request; and determining an output of the neural network machine learning model associated with the real-time inference request and one or more Shapley values associated with the output of the neural network machine learning model.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 526,230, filed on July 12, 2023, the disclosure of which is incorporated herein by reference in its entirety. Background Art 1. Technical Field

[0003] The present disclosure generally relates to analysis of machine learning models and, in some specific embodiments or aspects, to methods, systems, and computer program products for implementing a model agnostic framework to provide Shapley values associated with machine learning models.

[0004] 2. Technical considerations

[0005] Model interpretability (e.g., model interpretability or machine learning model transparency) can refer to the concept of being able to understand a machine learning model. In some instances, model interpretability can include machine learning explanations, which are a set of views of the model function that help users understand the results predicted by the machine learning model. Some methods for providing model explanations can include logistic regression coefficients, LIME, Shapley value techniques (e.g., QII, SHAP), and integrated gradient explanations.

[0006] Techniques based on Shapley values can be algorithmically interpretable and / or model-agnostic. Techniques based on Shapley values assume that the model's internals are inaccessible and can be applied to any model type. Techniques based on Shapley values can involve a core algorithm that can be applied to any input but can be used to explain the constituent features of a machine learning model. Other Shapley value-based interpretations can be used to determine local and / or global model reasoning for various model outputs (e.g., probability, regression, classification results, etc.).

[0007] The Shapley value can be a value derived by allocating credit for the output of a machine learning model across the input features that produced the output using fair distribution results from cooperative game theory. In some instances, the Shapley value can be calculated by carefully perturbing the input features and observing how the changes in the input features correspond to the final model predictions. The Shapley value for a given feature can then be calculated as the average marginal contribution to the final model prediction (e.g., the overall model score).

[0008] However, current model interpretability techniques, such as SHAP, may not be able to simultaneously obtain outputs (e.g., model scores) and model explanations. Furthermore, such techniques may require significant resources to provide explanations and may require significant memory. Furthermore, such techniques may only support machine learning models written in specific languages, such as SHAP's requirement for machine learning models written in PyTorch or TensorFlow. Summary of the Invention

[0009] Thus, improved methods, systems, and computer program products are provided for implementing a model-agnostic framework for providing Shapley values associated with machine learning models.

[0010] According to a non-limiting embodiment or aspect, a method for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model is provided, comprising: receiving an executable file for a neural network machine learning model; converting a format of the executable file for the neural network machine learning model to a model-agnostic format to provide a model-agnostic format file for the neural network machine learning model; parsing the model-agnostic format file for the neural network machine learning model, wherein parsing the model-agnostic format file for the neural network machine learning model comprises: storing a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving a real-time inference request for the neural network machine learning model; determining an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determining one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0011] In some non-limiting embodiments or aspects, the method further includes generating a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and reference input data provided to the neural network machine learning model.

[0012] In some non-limiting embodiments or aspects, generating a reverse symbolic graph associated with the neural network machine learning model includes generating the reverse symbolic graph associated with the neural network machine learning model based on a loss function for the neural network machine learning model.

[0013] In some non-limiting embodiments or aspects, wherein the forward sign graph includes a plurality of nodes and a plurality of edges, and wherein generating a reverse sign graph associated with the neural network machine learning model includes: calculating gradients between adjacent nodes in the plurality of nodes of the forward sign graph; and generating the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph.

[0014] In some non-limiting embodiments or aspects, generating a reverse symbolic graph associated with the neural network machine learning model includes generating the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0015] In some non-limiting embodiments or aspects, determining one or more Shapley values associated with an output of the neural network machine learning model includes applying an automatic differentiation algorithm to a reverse sign graph.

[0016] In some non-limiting embodiments or aspects, the method further includes determining a fraud detection score based on an output of the neural network machine learning model, wherein one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

[0017] According to a non-limiting embodiment or aspect, a system for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model is provided, comprising at least one processor configured to: receive an executable file for a neural network machine learning model; convert a format of the executable file for the neural network machine learning model to a model-agnostic format to provide a model-agnostic format file for the neural network machine learning model; parse the model-agnostic format file for the neural network machine learning model, wherein when parsing the model-agnostic format file for the neural network machine learning model, the at least one processor is configured to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generate a forward sign graph associated with the neural network machine learning model; and generate a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receive a real-time inference request for the neural network machine learning model; determine an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determine one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0018] In some non-limiting embodiments or aspects, at least one processor is further configured to generate a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and reference input data provided to the neural network machine learning model.

[0019] In some non-limiting embodiments or aspects, when generating a reverse symbolic graph associated with a neural network machine learning model, at least one processor is configured to: generate the reverse symbolic graph associated with the neural network machine learning model based on a loss function for the neural network machine learning model.

[0020] In some non-limiting embodiments or aspects, wherein a forward sign graph includes a plurality of nodes and a plurality of edges, and wherein when generating a reverse sign graph associated with a neural network machine learning model, at least one processor is configured to: calculate gradients between adjacent nodes in the plurality of nodes of the forward sign graph; and generate the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph.

[0021] In some non-limiting embodiments or aspects, wherein when generating the reverse symbolic graph associated with the neural network machine learning model, at least one processor is configured to: generate the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0022] In some non-limiting embodiments or aspects, wherein when determining one or more Shapley values associated with an output of the neural network machine learning model, at least one processor is configured to: apply an automatic differentiation algorithm to the inverse sign graph.

[0023] In some non-limiting embodiments or aspects, the at least one processor is further configured to: determine a fraud detection score based on an output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

[0024] According to a non-limiting embodiment or aspect, a computer program product for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model is provided, comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium containing program instructions that, when executed by at least one processor, cause the at least one processor to: receive an executable file for a neural network machine learning model; convert a format of the executable file for the neural network machine learning model to a model-agnostic format to provide the neural network machine learning model with an agnostic model format file; parse the agnostic model format file for the neural network machine learning model, wherein the program instructions causing the at least one processor to parse the agnostic model format file for the neural network machine learning model cause the at least one processor to: convert the neural network The method includes storing a plurality of intermediate weights and a plurality of reference outputs of a neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving a real-time inference request for the neural network machine learning model; determining, using the neural network machine learning model, an output of the neural network machine learning model associated with the real-time inference request; and determining one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0025] In some non-limiting embodiments or aspects, the program instructions further cause the at least one processor to generate a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and reference input data provided to the neural network machine learning model.

[0026] In some non-limiting embodiments or aspects, the program instructions that cause at least one processor to generate a reverse symbolic graph associated with a neural network machine learning model cause the at least one processor to: generate a reverse symbolic graph associated with the neural network machine learning model based on a loss function for the neural network machine learning model.

[0027] In some non-limiting embodiments or aspects, a forward sign graph includes a plurality of nodes and a plurality of edges, and wherein the program instructions that cause the at least one processor to generate a reverse sign graph associated with the neural network machine learning model cause the at least one processor to: calculate gradients between adjacent nodes in the plurality of nodes of the forward sign graph; and generate the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph.

[0028] In some non-limiting embodiments or aspects, the program instructions that cause at least one processor to generate a reverse symbolic graph associated with a neural network machine learning model cause the at least one processor to: generate the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0029] In some non-limiting embodiments or aspects, the program instructions that cause at least one processor to determine one or more Shapley values associated with an output of a neural network machine learning model cause the at least one processor to: apply an automatic differentiation algorithm to the inverse sign graph.

[0030] In some non-limiting embodiments or aspects, the program instructions further cause the at least one processor to: determine a fraud detection score based on an output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

[0031] Additional non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0032] Clause 1: A computer-implemented method comprising: receiving, with at least one processor, an executable file for a neural network machine learning model; converting, with at least one processor, a format of the executable file for the neural network machine learning model into a model-agnostic format to provide the neural network machine learning model with an agnostic model format file; parsing, with at least one processor, the agnostic model format file for the neural network machine learning model, wherein parsing the agnostic model format file for the neural network machine learning model comprises: storing, with at least one processor, a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on a plurality of intermediate weights and a plurality of reference outputs provided to the neural network machine learning model. The invention also provides a method for processing a real-time inference request for the neural network machine learning model; generating, using at least one processor, a forward sign graph associated with the neural network machine learning model; and generating, using at least one processor, a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving, using at least one processor, a real-time inference request for the neural network machine learning model; determining, using at least one processor, an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determining, using at least one processor, one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0033] Clause 2: The computer-implemented method of clause 1, further comprising generating a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

[0034] Clause 3: A computer-implemented method according to clause 1 or 2, wherein generating the reverse symbolic graph associated with the neural network machine learning model includes: generating the reverse symbolic graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0035] Clause 4: A computer-implemented method according to any one of clauses 1 to 3, wherein the forward sign graph includes a plurality of nodes and a plurality of edges, and wherein generating the reverse sign graph associated with the neural network machine learning model includes: calculating gradients between adjacent nodes in the plurality of nodes of the forward sign graph; and generating the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph.

[0036] Clause 5: A computer-implemented method according to any one of clauses 1 to 4, wherein generating the reverse symbolic graph associated with the neural network machine learning model includes: generating the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0037] Clause 6: The computer-implemented method of any one of clauses 1 to 5, wherein determining the one or more Shapley values associated with the output of the neural network machine learning model comprises applying an automatic differentiation algorithm to the inverse sign graph.

[0038] Clause 7: A computer-implemented method according to any one of clauses 1 to 6, further comprising: determining a fraud detection score based on the output of the neural network machine learning model, wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

[0039] Clause 8: A system comprising: at least one processor configured to: receive an executable file for a neural network machine learning model; convert a format of the executable file for the neural network machine learning model to a model-agnostic format to provide an agnostic model format file for the neural network machine learning model; parse the agnostic model format file for the neural network machine learning model, wherein when parsing the agnostic model format file for the neural network machine learning model, the at least one processor is configured to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are stored in a cache memory location. The plurality of reference outputs are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving a real-time inference request for the neural network machine learning model; determining an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determining one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0040] Clause 9: The system of clause 8, wherein the at least one processor is further configured to generate a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

[0041] Clause 10: The system of clause 8 or 9, wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: generate the reverse symbolic graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0042] Clause 11: A system according to any one of clauses 8 to 10, wherein the forward sign graph includes multiple nodes and multiple edges, and wherein when generating the reverse sign graph associated with the neural network machine learning model, the at least one processor is configured to: calculate gradients between adjacent nodes in the multiple nodes of the forward sign graph; and generate multiple nodes and multiple edges of the reverse sign graph based on the gradients between adjacent nodes in the multiple nodes of the forward sign graph.

[0043] Clause 12: The system of any one of clauses 8 to 11, wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: generate the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0044] Clause 13: The system of any one of clauses 8 to 12, wherein when determining the one or more Shapley values associated with the output of the neural network machine learning model, the at least one processor is configured to: apply an automatic differentiation algorithm to the inverse symbolic graph.

[0045] Clause 14: A system according to any one of clauses 8 to 13, wherein the at least one processor is further configured to: determine a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data contained in the real-time inference request that affect the fraud detection score.

[0046] Clause 15: A computer program product comprising at least one non-transitory computer-readable medium containing program instructions that, when executed by at least one processor, cause the at least one processor to: receive an executable file for a neural network machine learning model; convert a format of the executable file for the neural network machine learning model to a model-agnostic format to provide an agnostic model format file for the neural network machine learning model; parse the agnostic model format file for the neural network machine learning model, wherein the program instructions that cause the at least one processor to parse the agnostic model format file for the neural network machine learning model cause the at least one processor to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory; The invention relates to a method for storing a neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving a real-time inference request for the neural network machine learning model; determining an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determining one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0047] Clause 16: The computer program product of clause 15, wherein the program instructions further cause the at least one processor to: generate a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

[0048] Clause 17: A computer program product according to clause 15 or 16, wherein the program instructions that cause the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: generate the reverse symbolic graph associated with the neural network machine learning model based on the loss function for the neural network machine learning model.

[0049] Clause 18: A computer program product according to any one of clauses 15 to 17, wherein the forward sign graph comprises a plurality of nodes and a plurality of edges, and wherein the program instructions that cause the at least one processor to generate the reverse sign graph associated with the neural network machine learning model cause the at least one processor to: calculate gradients between adjacent nodes of the plurality of nodes of the forward sign graph; and generate the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes of the plurality of nodes of the forward sign graph.

[0050] Clause 19: A computer program product according to any one of clauses 15 to 18, wherein the program instructions causing the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: generate the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0051] Clause 20: A computer program product according to any one of clauses 15 to 19, wherein the program instructions that cause the at least one processor to determine the one or more Shapley values associated with the output of the neural network machine learning model cause the at least one processor to: apply an automatic differentiation algorithm to the inverse symbolic graph.

[0052] Clause 21: A computer program product according to any one of clauses 15 to 20, wherein the program instructions further cause the at least one processor to: determine a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data contained in the real-time inference request that affect the fraud detection score.

[0053] These and other features and characteristics of the present disclosure, as well as the methods of operation and function of the related structural elements and combinations of parts, and the economies of manufacture, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts throughout the several views. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Additional advantages and details of the present disclosure are explained in more detail below with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0055] Figure 1is a diagram of a non-limiting embodiment or aspect of an environment in which the systems, apparatus, products, devices, and / or methods described herein may be implemented according to the principles of the present disclosure;

[0056] Figure 2 is a flowchart of a non-limiting embodiment or aspect of a process for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model;

[0057] Figures 3A-3D is a schematic diagram of an exemplary embodiment of a system and / or method for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model, according to some non-limiting embodiments or aspects;

[0058] Figure 4 is a diagram of an exemplary environment in which systems, methods, and / or computer program products described herein may be implemented according to some non-limiting embodiments or aspects; and

[0059] Figure 5 According to some non-limiting embodiments or aspects Figure 1 and / or Figure 4 A schematic diagram of example components of one or more devices. DETAILED DESCRIPTION

[0060] For purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "transverse," "longitudinal," and their derivatives will be used relative to the orientation of the embodiments in the accompanying drawings. However, it should be understood that the embodiments may employ various alternative variations and step orders, except where expressly specified to the contrary. It should also be understood that the specific devices and processes illustrated in the accompanying drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Accordingly, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein should not be considered limiting.

[0061] Some non-limiting embodiments or aspects may be described herein in conjunction with a threshold value. As used herein, satisfying a threshold value may refer to a value greater than a threshold value, more than a threshold value, higher than a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, lower than a threshold value, less than or equal to a threshold value, equal to a threshold value, etc.

[0062] As used herein, the aspects, components, elements, structures, actions, steps, functions, instructions, etc. should not be understood as being critical or necessary unless explicitly described as such. Moreover, as used herein, the article "one" is intended to include one or more projects and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, a combination of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of wishing only one project, the term "one" or similar language is used. Moreover, as used herein, the term "having" etc. is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "at least partially based on". In addition, the mention of the action of "based on" a condition may refer to the action being "in response to" the condition. For example, in some non-limiting embodiments or aspects, the phrases "based on" and "in response to" may refer to the condition of automatically triggering an action (e.g., the specific operation of an electronic device such as a computing device, a processor, etc.).

[0063] As used herein, the term "acquirer" may refer to an entity that is licensed and / or authorized by a transaction service provider to initiate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. Transactions that an acquirer may initiate may include payment transactions (e.g., purchases, original letter of credit transactions (OCTs), account funding transactions (AFTs), etc.). In some non-limiting embodiments or aspects, the acquirer may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer, such as a server computer executing one or more software applications.

[0064] As used herein, the term "account identifier" may include one or more primary account numbers (PANs), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier that is used as a replacement or substitute identifier for an original account identifier, such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) such that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with multiple tokens used for different individuals or purposes.

[0065] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transfer, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). When one unit (e.g., a device, a system, a component of a device or system, a combination thereof, etc.) communicates with another unit, it means that the unit is able to directly or indirectly receive information from the other unit and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, etc.) that is wired and / or wireless in nature. In addition, although the transmitted information may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, a first unit may communicate with a second unit even if the first unit passively receives information and does not actively transmit the information to the second unit. As another example, a first unit may communicate with a second unit if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet, etc.) containing data. It should be understood that many other arrangements are possible.

[0066] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components to receive, process, and output data, such as a processor, a display, memory, an input device, a network interface, and the like. A computing device may be a mobile device. By way of example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0067] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing for multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that various other arrangements are possible. In addition, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) communicating directly or indirectly in a network environment may constitute a "system."

[0068] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processors, servers, client devices, software applications, components of such devices, etc.). As used herein, references to "device," "server," "processor," etc. may refer to a previously described device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and claims, a first device, first server, or first processor that is recited as performing a first step or a first function may refer to the same or a different device, server, or processor that is recited as performing a second step or a second function.

[0069] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide a customer with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with the customer. The account identifier may be embodied on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0070] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The terms "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0071] As used herein, the term "payment device" may refer to an electronic payment device, a portable financial device (e.g., a payment card, such as a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a keychain device or pendant, a radio frequency identification (RFID) transponder, a retailer discount or loyalty card, a cellular phone, an electronic wallet mobile application, a PDA, a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory for storing information (e.g., an account identifier, an account holder's name, etc.).

[0072] As used herein, a "point of sale (POS) device" may refer to one or more devices that can be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include a peripheral device, a card reader, a scanning device (e.g., a code scanner), Communication receiver, near field communication (NFC) receiver, RFID receiver and / or other contactless transceiver or receiver, contact-based receiver, payment terminal, etc. As used herein, a "point of sale (POS) system" may refer to one or more client devices and / or peripheral devices used by a merchant to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers that are configured to process online payment transactions through a web page, mobile application, etc.

[0073] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and an issuer organization. For example, a transaction service provider may include, for example , or any other entity that processes transactions. The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server that executes one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0074] Non-limiting embodiments or aspects of the disclosed subject matter are directed to methods, systems, and computer program products for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model. In some non-limiting embodiments or aspects, a model interpretation system may receive a file (e.g., an executable file) for a neural network machine learning model, convert the format of the file for the neural network machine learning model to a model-agnostic format (e.g., an Open Neural Network Exchange (ONNX) format), provide a model-agnostic file (e.g., an ONNX file) for the neural network machine learning model, and parse the model-agnostic file for the neural network machine learning model. In some non-limiting embodiments or aspects, when parsing the model-agnostic file for the neural network machine learning model, the model interpretation system may store intermediate weights and multiple reference outputs of the neural network machine learning model in a cache memory location, wherein the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model, generate a forward sign graph associated with the neural network machine learning model, and generate a reverse sign graph associated with the neural network machine learning model based on the forward sign graph. In some non-limiting embodiments or aspects, the model interpretation system may receive a real-time inference request for a neural network machine learning model, determine an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model, and determine one or more Shapley values associated with the output of the neural network machine learning model based on the reverse sign graph and intermediate weights and multiple reference outputs of the neural network machine learning model stored in a cache memory location.

[0075] In some non-limiting embodiments or aspects, the model interpretation system may generate a loss function for the neural network machine learning model based on a difference between an output of the forward sign graph and reference input data provided to the neural network machine learning model. In some non-limiting embodiments or aspects, when generating a reverse sign graph associated with the neural network machine learning model, the model interpretation system may generate the reverse sign graph associated with the neural network machine learning model based on the loss function for the neural network machine learning model.

[0076] In some non-limiting embodiments or aspects, a forward sign graph includes a plurality of nodes and a plurality of edges, and when generating a reverse sign graph associated with a neural network machine learning model, the model interpretation system may calculate gradients between adjacent nodes in the plurality of nodes of the forward sign graph and generate the plurality of nodes and the plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph.

[0077] In some non-limiting embodiments or aspects, when generating a reverse symbolic graph associated with a neural network machine learning model, the model interpretation system may generate the reverse symbolic graph associated with the neural network machine learning model to include a linear operator and / or a nonlinear operator.

[0078] In some non-limiting embodiments or aspects, the model interpretation system may apply an automatic differentiation algorithm to the inverse sign graph. In some non-limiting embodiments or aspects, the model interpretation system may determine a fraud detection score based on an output of the neural network machine learning model, and one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

[0079] In this way, the model interpretation system can be able to provide output (e.g., a model score indicating the accuracy of the machine learning model with respect to inference) and model explanations simultaneously (e.g., near simultaneously) and in real time (e.g., at or near the time of operation of the model interpretation system). Furthermore, the model interpretation system can reduce the amount of resources required to provide explanations and provide faster response times, while providing a framework that is independent of the type of framework (e.g., language type) used to initially prepare the machine learning model.

[0080] For illustrative purposes, in the following description, although the presently disclosed subject matter is described with respect to methods, systems, and computer program products for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model, wherein the Shapley values provide an explanation of the output of the machine learning model by attributing the contribution of each feature to the final output (e.g., prediction, model score, etc.), so as to gain a deeper understanding of the features that have an impact on the output of the machine learning model and help understand and interpret the behavior of the machine learning model, those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings, such as prediction, regression, classification, fraud prevention, authorization, authentication, feature selection, and the like.

[0081] For purposes of illustration, in the following description, although the presently disclosed subject matter is described with respect to methods, systems, and computer program products for a large-scale graph transformer machine learning model network architecture that can be used in association with providing recommendations, those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings and / or for making determinations (e.g., prediction, classification, regression, etc.), such as for fraud detection / prevention, authorization, authentication, identification, feature selection, payment processing, etc.

[0082] Now refer to Figure 1 , Figure 1 is a diagram of an example system 100 in which the devices, systems, and / or methods described herein may be implemented. Figure 1 As shown in FIG, system 100 includes a model interpretation system 102, a machine learning (ML) model management database 104, a user device 106, and a communication network 108. The model interpretation system 102, the ML model management database 104, and / or the user device 106 can be interconnected (e.g., connected to communicate) via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection.

[0083] The model interpretation system 102 may include one or more devices capable of receiving information from and / or transmitting information to the ML model management database 104 and / or the user device 106 via the communication network 108 (e.g., directly via a wired or wireless communication connection, indirectly via the communication network 108, etc.). For example, the model interpretation system 102 may include a server, a server cluster, a cloud platform, and / or other similar devices. In some non-limiting embodiments or aspects, the model interpretation system 102 may be associated with a transaction service provider system. For example, the model interpretation system 102 may be operated by the transaction service provider system. In another example, the model interpretation system 102 may be a component of the user device 106. In another example, the model interpretation system 102 may include the ML model management database 104. In some non-limiting embodiments or aspects, the model interpretation system 102 may communicate with a data storage device (e.g., the ML model management database 104), which may be local or remote to the model interpretation system 102. In some non-limiting embodiments or aspects, the model interpretation system 102 may be capable of receiving information from a data store, storing information in a data store, transmitting information to a data store, and / or searching for information stored in a data store.

[0084] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., train, validate, retrain, etc.), store, and / or implement (e.g., operate, provide input to, and / or output from, one or more machine learning models, etc.) one or more machine learning models. For example, the model interpretation system 102 may generate one or more machine learning models by fitting (e.g., validating, testing, etc.) one or more machine learning models against data used for training (e.g., training data). In some non-limiting embodiments or aspects, the model interpretation system 102 may generate, store, and / or implement one or more machine learning models, which are provided to a production environment (e.g., a runtime environment, a real-time environment, etc.) for providing inferences (e.g., safety inferences) based on data inputs in live scenarios (e.g., real-time scenarios). Additionally or alternatively, the model interpretation system 102 may generate, store, and / or implement one or more machine learning models, which are provided to a non-production environment (e.g., an offline environment, a training environment, etc.) for providing inferences based on data inputs in non-live scenarios. In some non-limiting embodiments or aspects, the model interpretation system 102 may communicate with a data store (the ML model management database 104 ), which may be local or remote to the model interpretation system 102 .

[0085] The ML model management database 104 may include one or more devices capable of receiving information from the model interpretation system 102 and / or the user device 106 and / or transmitting information to the model interpretation system and / or the user device (e.g., directly via a wired or wireless communication connection, indirectly via the communication network 108, etc.). For example, the ML model management database 104 may include a server, a server cluster, a desktop computer, a portable computer, a mobile device, and / or other similar devices. In some non-limiting embodiments or aspects, the ML model management database 104 may include a data storage device. In some non-limiting embodiments or aspects, the ML model management database 104 may be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device. In some non-limiting embodiments or aspects, the ML model management database 104 may be part of the model interpretation system 102 and / or part of the same system as the model interpretation system 102.

[0086] The user device 106 may include one or more devices capable of receiving information from and / or transmitting information to the model interpretation system 102 and / or the ML model management database 104 (e.g., directly via a wired or wireless communication connection, indirectly via the communication network 108, etc.). For example, the user device 106 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other similar devices. Additionally or alternatively, the user device 106 may include a device capable of receiving information from and / or transmitting information to other user devices (e.g., directly via a wired or wireless communication connection, indirectly via the communication network 108, etc.). In some non-limiting embodiments or aspects, the user device 106 may be part of the model interpretation system 102 and / or part of the same system as the model interpretation system 102. For example, the model interpretation system 102, the ML model management database 104, and the user device 106 may all be a single system and / or a single computing device (and / or a portion thereof).

[0087] The communication network 108 may include one or more wired and / or wireless networks. For example, the communication network 108 may include a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, a cloud computing network, etc., and / or a combination of some or all of these or other types of networks.

[0088] Figure 1 The number and arrangement of the systems and devices shown in are provided as examples. Figure 1 There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged differently than the systems and / or devices shown in FIG. Furthermore, a single system and / or device may be implemented. Figure 1 Two or more systems or devices shown in, or Figure 1 The single system or device shown in FIG100 may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0089] Now refer to Figure 2, a flowchart of a process 200 for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model, according to some non-limiting embodiments or aspects, is shown. In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., entirely, partially, etc.) by the model interpretation system 102 (e.g., one or more devices of the model interpretation system 102). In some non-limiting embodiments or aspects, one or more of the steps of process 200 may be performed (e.g., entirely, partially, etc.) by another device or group of devices that is separate from or includes the model interpretation system 102 (e.g., one or more devices of the model interpretation system 102), the ML model management database 104, and / or the user device 106. Figure 2 The steps shown in the foregoing are for exemplary purposes only. It will be appreciated that in some non-limiting embodiments or aspects, additional, fewer, different, and / or different orders of steps may be used. In some non-limiting embodiments or aspects, a step may be automatically performed in response to the execution and / or completion of a previous step.

[0090] like Figure 2 As shown in , at step 202, process 200 includes receiving a file for a machine learning model. For example, the model interpretation system 102 may receive a file for a machine learning model. In one example, the file may include an executable file for a machine learning model, such as a neural network machine learning model. In some non-limiting embodiments or aspects, the file for the machine learning model may have a format based on the type of machine learning framework used to develop the machine learning model (e.g., Keras, PyTorch, TensorFlow, Caffe, Matlab, etc.). In some non-limiting embodiments or aspects, the model interpretation system 102 may receive data associated with the machine learning model, which may include a file for the machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may receive data from the ML model management database 104, the user device 106, and / or another system or device.

[0091] like Figure 2As shown in , at step 204, process 200 includes converting the format of a file for a machine learning model to provide an agnostic model format file for the machine learning model. For example, the model interpretation system 102 may convert the format of a file for a machine learning model to an agnostic model format to provide an agnostic model format file for the machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may convert the format of an executable file for a machine learning model (e.g., a neural network machine learning model) to an ONNX format to provide an ONNX file for the machine learning model. Additionally or alternatively, the model interpretation system 102 may convert the format of a file for a machine learning model to a standardized model format (e.g., a Predictive Model Markup Language (PMML) format, a Portable Analysis Format (PFA) format, a TensorFlowSavedModel format, a Keras HDF5 format, a Core ML format, an MXNet model format, a Caffe model format, etc.) to provide the machine learning model with a standardized model file to be used as an agnostic model format file.

[0092] like Figure 2 As shown in , at step 206, process 200 includes parsing a model-agnostic format file for a machine learning model to provide a symbolic graph associated with the machine learning model. For example, the model interpretation system 102 may parse a model-agnostic format file (e.g., an ONNX file) for a machine learning model to provide a forward symbolic graph and / or a reverse symbolic graph associated with the machine learning model. In some non-limiting embodiments or aspects, the symbolic graph associated with the machine learning model may include a high-level representation of the computational flow of the machine learning model. The symbolic graph may include multiple nodes and multiple edges to define the structure (e.g., architecture) and / or operations of the machine learning model. In some non-limiting embodiments or aspects, each node in the symbolic graph may represent an operation (e.g., addition, multiplication, convolution, etc.), and each edge may represent the flow of data between the operations.

[0093] In some non-limiting embodiments or aspects, a forward sign graph associated with a machine learning model may include a type of computational graph that represents the series of operations required to compute the output of the machine learning model from the inputs provided to the machine learning model. The forward sign graph may define the flow of data through the machine learning model during the forward propagation, where the input data is processed to produce an output (e.g., a prediction, a model score, etc.). In some non-limiting embodiments or aspects, the nodes of the forward sign graph may represent operations and / or layers in the machine learning model and may include mathematical functions, activation functions, layers (e.g., convolutional layers, fully connected layers, etc.), and / or other processing steps. In some non-limiting embodiments or aspects, the edges of the forward sign graph may represent the flow of data between nodes, and each edge may represent the output of an input passed from one node to another node. In some non-limiting embodiments or aspects, the forward sign graph may begin with one or more input nodes representing the raw data provided to the machine learning model and end with one or more output nodes representing the outputs. In some non-limiting embodiments or aspects, the forward sign graph may be deterministic, such that given the same input, the forward sign graph will produce the same output.

[0094] In some non-limiting embodiments or aspects, a reverse symbolic graph associated with a machine learning model may represent the series of operations required to calculate the gradients of model parameters during backpropagation. In some non-limiting embodiments or aspects, the reverse symbolic graph may define how gradients propagate back through the machine learning model to update weights. In some non-limiting embodiments or aspects, the nodes of the reverse symbolic graph may represent the gradient calculations used for each operation in the forward pass. Nodes may include loss function gradients, intermediate activation gradients, and / or model parameter gradients. In some non-limiting embodiments or aspects, the edges of the reverse symbolic graph may represent the flow of gradients between nodes. Each edge may represent a gradient from one node to a previous node that contributed to the calculation of the gradient. In some non-limiting embodiments or aspects, the reverse symbolic graph may illustrate flow in the opposite direction of the forward symbolic graph. The reverse symbolic graph may start at a node with a loss and propagate the gradient back to one or more input nodes. In some non-limiting embodiments or aspects, each node of the reverse symbolic graph may correspond to a partial derivative of the loss with respect to one or more variables involved in the forward pass, and the partial derivatives may be used by the model interpretation system 102 to update the model parameters of the machine learning model.

[0095] In some non-limiting embodiments or aspects, the forward sign graph and / or the reverse sign graph may include a plurality of nodes (e.g., vertices) and a plurality of edges. In some non-limiting embodiments or aspects, the forward sign graph and / or the reverse sign graph may include a set of nodes (e.g., a set of at least 5, 10, 15, 30, 50, 100, 200, 300, etc., or more nodes) and / or a set of edges (e.g., a set of at least 5, 10, 15, 30, 50, 100, 200, 300, etc., or more edges).

[0096] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of intermediate weights and / or a plurality of reference outputs of a machine learning model (e.g., a neural network machine learning model) based on reference input data provided to the machine learning model. For example, the model interpretation system 102 may provide the reference input data as input to the machine learning model, and the machine learning model may provide a plurality of reference outputs of the machine learning model based on the input. When the model parameters of the machine learning model are updated based on the forward propagation of the reference input data, the plurality of intermediate weights may be generated during the backpropagation.

[0097] In some non-limiting embodiments or aspects, the model interpretation system 102 may receive a dataset (e.g., a training dataset, a reference dataset, etc.) containing reference input data. For example, the model interpretation system 102 may receive the dataset from the ML model management database 104. In some non-limiting embodiments or aspects, the reference input data may be associated with one or more entities of a population of entities (e.g., users, account holders, merchants, issuers, items provided by an entity, etc.). In some non-limiting embodiments or aspects, the reference input data may include multiple data instances associated with multiple features. In some non-limiting embodiments or aspects, the multiple data instances of the graph data may represent multiple interactions (e.g., transactions, such as electronic payment transactions) involving one or more entities of the population. In some examples, the reference input data may include a large number of data instances, such as 100 data instances, 500 data instances, 1,000 data instances, 5,000 data instances, 10,000 data instances, 25,000 data instances, 50,000 data instances, 100,000 data instances, 1,000,000 data instances, and the like.

[0098] In some non-limiting embodiments or aspects, each data instance may include transaction data associated with a transaction. In some non-limiting embodiments or aspects, the transaction data may include multiple transaction parameters associated with the electronic payment transaction. In some non-limiting embodiments or aspects, multiple features may represent multiple transaction parameters. In some non-limiting embodiments or aspects, the plurality of transaction parameters may include electronic wallet card data associated with an electronic card (e.g., an electronic credit card, an electronic debit card, an electronic membership card, etc.), decision data associated with a decision (e.g., a decision to approve or deny a transaction authorization request), authorization data associated with an authorization response (e.g., an approved spending limit, an approved transaction value, etc.), a PAN, an authorization code (e.g., a personal identification number (PIN)), etc.), data associated with a transaction amount (e.g., an approved limit, a transaction value, etc.), data associated with a transaction date and time, data associated with a currency conversion rate, data associated with a merchant type (e.g., a merchant category code indicating a type of merchandise such as groceries, fuel, etc.), data associated with an acquirer country, data associated with an identifier of a country associated with the PAN, data associated with a response code, data associated with a merchant identifier (e.g., a merchant name, a merchant location, etc.), data associated with a currency type corresponding to funds stored associated with the PAN, etc.

[0099] In some non-limiting embodiments or aspects, the model interpretation system 102 may store the intermediate weights and / or multiple reference outputs of the machine learning model. For example, the model interpretation system 102 may store the intermediate weights and / or multiple reference outputs of the machine learning model in a cache memory location (e.g., a cache memory location of the model interpretation system 102). In this way, the model interpretation system 102 may be able to access the intermediate weights and / or multiple reference outputs of the machine learning model stored in the cache memory location more quickly than if the intermediate weights and / or multiple reference outputs of the machine learning model were stored in another location.

[0100] In some non-limiting embodiments or aspects, the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model. In the above example, the model interpretation system 102 can generate a forward sign graph associated with the neural network machine learning model and a reverse sign graph associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 can generate a reverse sign graph based on the forward sign graph.

[0101] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a loss function for the machine learning model based on the difference between the output of the forward sign graph and the reference input data provided to the machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse sign graph associated with the neural network machine learning model based on the loss function for the machine learning model.

[0102] In some non-limiting embodiments or aspects, the model interpretation system 102 may calculate gradients associated with a forward sign graph. For example, the model interpretation system 102 may calculate gradients between adjacent nodes in the plurality of nodes of the forward sign graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges of a reverse sign graph based on the gradients associated with the forward sign graph. For example, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse sign graph associated with a neural network machine learning model to include a linear operator and / or a nonlinear operator.

[0103] In some non-limiting embodiments or aspects, the model interpretation system 102 can apply an automatic differentiation algorithm to the inverse symbolic graph. For example, the model interpretation system 102 can apply an automatic differentiation algorithm to the inverse symbolic graph to optimize (e.g., simplify) the inverse symbolic graph, which can then be used to generate one or more Shapley values.

[0104] like Figure 2 As shown in , at step 208, process 200 includes receiving a real-time inference request for the machine learning model. For example, the model interpretation system 102 may receive a real-time inference request for the machine learning model. In some non-limiting embodiments or aspects, the real-time inference request may be based on a task (e.g., a classification task) of the machine learning model. For example, the real-time inference request may be based on a request to determine whether a transaction (e.g., a transaction involving a user of the user device 106) is fraudulent.

[0105] like Figure 2 As shown in , at step 210, process 200 includes determining an output of a machine learning model associated with a real-time inference request and one or more Shapley values associated with the output. For example, the model interpretation system 102 can determine an output of a machine learning model associated with a real-time inference request and / or one or more Shapley values associated with the output.

[0106] In some non-limiting embodiments or aspects, the model interpretation system 102 may use a machine learning model to determine an output of the machine learning model associated with an input included in a real-time inference request. For example, the model interpretation system 102 may generate a score (e.g., a model score, a prediction score, etc.) based on the input provided to the machine learning model. In such examples, the model interpretation system 102 may generate a score based on the input provided to the machine learning model to generate the score included with the inference request. In some non-limiting embodiments or aspects, the score of the input (e.g., a data instance) may be equal to the average model score (e.g., the average model score of all inputs in the plurality of inputs) added to the sum of the Shapley values of each feature in the plurality of features included in the input.

[0107] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., determine) a score associated with an inference task based on an output of a machine learning model, the output of the machine learning model being generated based on input data provided as input to the machine learning model (e.g., input data included in an inference request). In one example, the model interpretation system 102 may generate a fraud detection score based on the output of the machine learning model, and one or more Shapley values associated with the output of the machine learning model may include an indication of one or more features of the input data that affect the fraud detection score.

[0108] In some non-limiting embodiments or aspects, the model interpretation system 102 may determine one or more Shapley values associated with an output of the machine learning model based on an inverse sign graph of the machine learning model, a plurality of intermediate weights, and / or a plurality of reference outputs (e.g., a plurality of reference outputs of the machine learning model stored in a cache memory location). In some non-limiting embodiments or aspects, when determining the one or more Shapley values associated with an output of the neural network machine learning model, the model interpretation system 102 may apply an automatic differentiation algorithm to the inverse sign graph.

[0109] In some non-limiting embodiments or aspects, the model interpretation system 102 may perform an action, such as a fraud prevention program, a transaction authorization program, a recommendation program, etc., based on the output of the machine learning model and / or one or more Shapley values associated with the output. For example, the model interpretation system 102 may perform an action based on determining to perform the action after analyzing the output and / or one or more Shapley values associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 may perform a fraud prevention program associated with protecting an account of a user (e.g., a first entity, such as a user associated with the user device 106) based on the output of the machine learning model and / or one or more Shapley values associated with the output. For example, if the output of the machine learning model and / or one or more Shapley values associated with the output (e.g., one or more Shapley values associated with the output having a value indicating that the machine learning model correctly predicted that a fraud prevention program is necessary) indicates that a fraud prevention program is necessary, the model interpretation system 102 may perform the fraud prevention program associated with protecting the user's account. In such examples, if the output of the machine learning model and / or one or more Shapley values associated with the output (e.g., one or more Shapley values associated with the output having a value indicating that the machine learning model did not correctly predict that a fraud prevention program is necessary) indicates that a fraud prevention program is not necessary, the model interpretation system 102 may forgo executing the fraud prevention program associated with protecting the user's account.

[0110] In some non-limiting embodiments or aspects, the model interpretation system 102 may perform an action associated with the machine learning model based on the output of the machine learning model and / or one or more Shapley values associated with the output, such as a feature selection procedure, a training (e.g., retraining) procedure, an inference task (e.g., performing a real-time inference task, such as another real-time inference task), etc. For example, the model interpretation system 102 may perform an action associated with the machine learning model based on determining to perform the action after analyzing the output and / or one or more Shapley values associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 may perform an action associated with the machine learning model based on the output of the machine learning model and / or one or more Shapley values associated with the output. For example, if the output of the machine learning model and / or one or more Shapley values associated with the output indicate that the action associated with the machine learning model is necessary, the model interpretation system 102 may perform a fraud prevention procedure associated with protecting a user's account. In such an example, if the output of the machine learning model and / or one or more Shapley values associated with the output indicate that the action associated with the machine learning model is not necessary, the model interpretation system 102 may forgo performing the action associated with the machine learning model.

[0111] Now refer to Figures 3A-3D, a schematic diagram of an embodiment 300 of a process (e.g., process 200) for implementing a model-agnostic framework to provide Shapley values associated with a machine learning model is shown. In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., entirely, partially, etc.) by the model interpretation system 102 (e.g., one or more devices of the model interpretation system 102). In some non-limiting embodiments or aspects, one or more of the steps of the process may be performed (e.g., entirely, partially, etc.) by another device or group of devices that is separate from or includes the model interpretation system 102 (e.g., one or more devices of the model interpretation system 102), the ML model management database 104, and / or the user device 106. As shown in embodiment 300, the Shapley value can be used to explain the difference between an output and a reference output based on the difference between the input and the corresponding reference input, which can be used to measure the importance of the target input to the output (e.g., prediction) of the machine learning model through backpropagation.

[0112] As Figure 3A As shown in FIG305 , the model interpretation system 102 may receive an executable file for a neural network machine learning model from the ML model management database 104. In some non-limiting embodiments or aspects, for a neural network machine learning model, t may represent the output of a neuron in an intermediate layer of the neural network machine learning model, and x0, x1, ..., x n can represent the input used to calculate t from the neuron.

[0113] The reference-from-difference Δt can be expressed as Δt=t-t0, where t0 is the reference input (eg, which may be selected based on domain knowledge and / or heuristics), and the model interpretation system 102 may convert the contribution fraction CΔx into i Δt is assigned to the constrained Δx i :

[0114]

[0115] where CΔx i Δt is the fraction of t attributed to x i The reference difference (difference-from-reference) is the reference difference amount.

[0116] The multiplier (e.g., derivative) can be defined as follows:

[0117]

[0118] Where Δx is the reference difference in input x, and Δt is the reference difference in output t. In some non-limiting embodiments or aspects, the multiplier can be used as a discrete version of the partial derivative because the contribution of Δx to Δt is divided by the input difference Δx. The chain rule for the multiplier can be defined as follows:

[0119]

[0120] where xi is the layer H used in the neural network machine learning model l The neuron input, and y0, y1, ... y n is used for layer H l The neuron output and the H l The analogy with partial derivatives allows calculation of the contribution of the neural network machine learning model output relative to the neural network machine learning model input via backpropagation. The Shapley value can be approximated by the average value according to the following formula:

[0121]

[0122] Where M is the final matrix calculated by the multiplier relative to the model input in backpropagation, X is the input, and R is the reference input. The present disclosure provides a method for implementing and accelerating the calculation of M in a model-agnostic framework (e.g., the ONNX ecosystem) for neural network machine learning models. In such a model-agnostic framework, gradient calculations can be adjusted for nonlinear operators (e.g., Sigmoid operator, MaxPooling operator, etc.), and raw gradient calculations can be used for linear operators (e.g., MatMul operator, convolution (Conv) operator, etc.).

[0123] As Figure 3A As further shown in FIG. 310 , the model interpretation system 102 may convert the format of the executable file for the neural network machine learning model into a model-agnostic format to provide a model-agnostic file. In some non-limiting embodiments or aspects, the model interpretation system 102 may convert the format of the executable file for the neural network machine learning model into an ONNX format to provide an ONNX file for the neural network machine learning model.

[0124] As Figure 3BAs shown in FIG315 , the model interpretation system 102 may parse a model-agnostic file for a neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may parse a model-agnostic format file (e.g., an ONNX file) for a neural network machine learning model to provide a forward symbol graph and / or a reverse symbol graph associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the forward symbol graph and / or the reverse symbol graph associated with the neural network machine learning model may include a high-level representation of the computational flow of the neural network machine learning model. The forward symbol graph and / or the reverse symbol graph may include multiple nodes (e.g., computational nodes) and multiple edges to define the structure (e.g., architecture) and / or operations of the neural network machine learning model. In some non-limiting embodiments or aspects, each node in the forward symbol graph and / or the reverse symbol graph may represent an operator (e.g., addition, multiplication, convolution, etc.), and each edge may represent the data flow between the operators.

[0125] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of intermediate weights and / or a plurality of reference outputs of the neural network machine learning model based on reference input data provided to the machine learning model. For example, the model interpretation system 102 may provide the reference input data as input to the neural network machine learning model, and the neural network machine learning model may provide a plurality of reference outputs of the neural network machine learning model based on the input. When the model parameters of the neural network machine learning model are updated based on the forward propagation of the reference input data, the plurality of intermediate weights may be generated during the backpropagation.

[0126] In some non-limiting embodiments or aspects, the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model. In the above example, the model interpretation system 102 can generate a forward sign graph associated with the neural network machine learning model and a reverse sign graph associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 can generate a reverse sign graph based on the forward sign graph.

[0127] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a loss function for the neural network machine learning model based on the difference between the output of the forward sign graph and the reference input data provided to the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse sign graph associated with the neural network machine learning model based on the loss function for the neural network machine learning model.

[0128] In some non-limiting embodiments or aspects, the model interpretation system 102 may calculate gradients associated with a forward sign graph. For example, the model interpretation system 102 may calculate gradients between adjacent nodes in the plurality of nodes of the forward sign graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges of a reverse sign graph based on the gradients associated with the forward sign graph. For example, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges of the reverse sign graph based on the gradients between adjacent nodes in the plurality of nodes of the forward sign graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse sign graph associated with a neural network machine learning model to include a linear operator and / or a nonlinear operator.

[0129] In some non-limiting embodiments or aspects, the model interpretation system 102 may first build a forward symbolic graph. In a forward symbolic graph, a node is linked to one or more other nodes because the output of a node can be an input to another node or the output of a neural network machine learning model (e.g., a model associated with an agnostic model format file, such as an ONNX model). From this, the model interpretation system 102 may construct a reverse graph, which may not yet include a reverse symbolic graph, but rather only include a graph structure with nodes that carry information about the nodes of the forward symbolic graph. The information in each node of the reverse graph may include the node itself, the number of neighbors of the current node and the neighbors of the node in the reverse graph, the incoming and / or outgoing gradients, and / or an optional argument indicating whether to pass gradients (e.g., passgrad) to indicate whether the input to the node is different from the input to the neural network machine learning model, which can be referred to as the model input.

[0130] Some operators, such as multiplication (Mul) and addition (Add), allow two inputs to differ from the model inputs. Other operators, such as matrix multiplication (MatMul) and general matrix multiplication (Gemm), may allow at most two inputs to vary when only one input differs from the model input. In some non-limiting embodiments or aspects, the model interpretation system 102 may determine whether the outflow gradient of the current node can be passed to the adjacent node via pass grad when constructing the reverse graph (e.g., using a neural network parser).

[0131] In some non-limiting embodiments or aspects, the model-agnostic framework may include multiple operators (e.g., hundreds of operators). In some non-limiting embodiments or aspects, the model interpretation system 102 may include gradients (e.g., for linear operators) or multipliers (e.g., for nonlinear operators) defined in the model-agnostic framework. In some non-limiting embodiments or aspects, the multiple operators may include Concat, Add, Mul, MatMul, Gemm, Sigmoid, ReLU, Softmax, Conv, MaxPool, AveragePool, GlobalAveragePool, Transpose, BatchNormalization, etc. When executing the forward sign graph, some of the resulting outputs of the gradient calculations may be stored in memory by the model interpretation system 102 for some operators. In this way, additional computations may be avoided when training a neural network machine learning model.

[0132] In some non-limiting embodiments or aspects, linear rules for linear operations may be used to calculate gradients, and rescale rules and / or reveal cancel rules may be used for non-linear operations to calculate multipliers. In some non-limiting embodiments or aspects, some operators that use forward sign graphs for gradient calculation and / or multiplier calculation include Concat, Mul, Matmul, Sigmoid, Maxpooling, GlobalMaxPooling, Avgpooling, and / or GlobalAvgPooling.

[0133] Concat can be a linear operator with no local gradients. In some non-limiting embodiments or aspects, the incoming gradients can be split and / or passed to subsequent nodes in the reverse path as part of the way the inputs to Concat are concatenated in the forward path. The effect of Mul can be nonlinear or linear, depending on whether the two inputs to the multiplication operation are differentiable with respect to the model inputs. If the two inputs to the multiplication operation are differentiable with respect to the model inputs, the model interpretation system 102 can use the appearance cancellation rule to calculate the adjusted gradient. Otherwise, the model interpretation system 102 can multiply the incoming gradient with the input to the multiplication operation that is not differentiable with respect to the model input to calculate the outgoing gradient. In the case of broadcasting, the smaller input can add the incoming gradient on the axis to which this input is broadcasted to the larger input of the multiplication operation.

[0134] Matmul is a linear operator, and the local gradient of Matmull is the transpose of the weight with respect to the input. Multiplying the local gradient with the incoming gradient provides the outgoing gradient to the successor in the reverse path. Conv is a linear operator that can be used to calculate the outgoing gradient for the Conv operation. Sigmoid is a nonlinear operation. The calculation of the adjusted gradient is defined as follows:

[0135]

[0136] Where σ(x) is the output of the Sigmoid, x is the input to some neuron for the data we want to explain, and r is the input to some neuron for the reference data. If xr < 1e-6 returns true, then the raw gradient of the Sigmoid can be used. Otherwise, the multiplier used for the Sigmoid can be used by the rescaling rule. Multiplying grad* with the incoming gradient provides the outgoing gradient. In some non-limiting embodiments or aspects, most activation functions use the same method to obtain grad*, except for Softmax, which uses an explicit cancellation rule.

[0137] Maxpooling is a nonlinear operation, and the adjusted gradient for Maxpooling can be defined as follows:

[0138] C=max(y x ,y r )

[0139] M x =(Cy r )×grad in

[0140] M r =(y x -C)×grad in

[0141]

[0142] where x and r are the inputs to the max pooling neuron for the input data and the reference input data, respectively, and y x and y r is the output of these neurons. C is y x Elements and y r The maximum value of the cross between elements. Pass in the gradient grad in Multiply by Cy r , to obtain the incoming gradient M of the cross-localization x Similarly, obtain M r If xr is less than 1e-7, the outgoing gradient is zero. Otherwise, the sum of the Maxpooling localization gradients with respect to x and r is divided by xr as the outgoing gradient The incoming gradient can be passed back to the neuron that achieved the maximum value, and when calculating the gradient for the Maxpooling operation, all other neurons have zero gradient. It should be noted that if the same neuron achieves the maximum value in different pooling windows, the gradients are accumulated. GlobalMaxPooling can include a special case of Maxpooling, whose pooling window size is the same as the input space. In addition, Avgpooling is a linear operation. In order to calculate the gradient of the input with respect to the Avgpooling operation, the incoming gradient can be evenly distributed to the positions within the pooling window, and if the two pooling windows overlap, the gradients can be accumulated. In addition, GlobalAvgPooling is a special case of Avgpooling, whose pooling window size is the same as the input space.

[0143] In some non-limiting embodiments or aspects, an automatic differentiation algorithm may be used to implement machine learning techniques, such as backpropagation (e.g., for training neural network machine learning models). In some non-limiting embodiments or aspects, the model interpretation system 102 may implement an automatic differentiation algorithm that performs a depth-first search (DFS) to identify all operators in the reverse path from the output to the input of the model and sums the partial gradients contributed by each operator. In some non-limiting embodiments or aspects, when using DFS, multiple types of gradient flows may be analyzed. For example, four types of gradient flows may include one2one, many2one, one2many, and many2many.

[0144] In a one2one type gradient flow, both the incoming gradient and the outgoing gradient have one branch, and the incoming gradient is multiplied by the local gradient (e.g., if any) to obtain the outgoing gradient. A one2one type gradient flow may contain an activation function, which is a typical operator of this type. If the operator has no local gradient, the incoming gradient is passed to the successor in the reverse path. In a many2one type gradient flow, there are multiple incoming gradient flows, but only one outgoing gradient flow. All incoming gradients are first summed, and then the sum is multiplied by the local gradient (e.g., if any) to obtain the outgoing gradient. In a one2many type gradient flow, there is one incoming gradient flow and multiple outgoing gradient flows. After multiplying the incoming gradient by the local gradient (e.g., if any), the outgoing gradient is split or distributed to the successor. A many2many type gradient flow is a combination of many2one and one2many.

[0145] In some non-limiting embodiments, the model interpretation system 102 may use a DFS algorithm to invert the forward sign graph to calculate one or more Shapley values. The procedure for calculating the Shapley value using DFS is provided below:

[0146] 1: Let S be the stack.

[0147] 2:S.push(N)

[0148] 3:Mark N as visited.

[0149] 4Define the difference-from-reference y x -y r as the loss grad in .{y is the output of model.}

[0150] 5:while S is not empty do

[0151] 6:C←S.pop()

[0152] 7:O,grad in ←F grad (C,G,grad in ){F grad is the function to compute gradients / multipliers for opera-tors.}

[0153] 8:Append O to L.

[0154] 9:for neighbor W of C in G do

[0155] 10:if W is not visited and it gets all gradient flows then

[0156] 11:S.push(W)

[0157] 12:Mark W as visited.

[0158] 13:end if

[0159] 14:end for

[0160] 15:end while

[0161] 16:return L

[0162] In the above program, the input includes a reverse graph G, a first computational node N, and the output includes a gradient node list L. In some non-limiting embodiments or aspects, DFS takes the reverse graph G and the first computational node N as input and returns a computational node list. In some non-limiting embodiments or aspects, the reverse graph G is obtained based on parsing an agnostic model format file for a neural network machine learning model, and N is the first computational node in the reverse path. Each node in G contains information for performing DFS, and the name of the access computational node is used to obtain the information. From lines 1-3, an empty stack is created, and N is pushed onto the stack, marking N as visited. Line 4 defines the loss y x -y r , to calculate the gradient with respect to the model input. The rest of the DFS algorithm details how to traverse all computation nodes in the reverse path. Function F grad Return to the computation node list O to compute the gradient of the access node C and the next node grad in line 7 in If a neighboring node W of C has not been visited and receives all incoming gradient flows, W is pushed onto the stack and marked as visited. In some non-limiting embodiments or aspects, the use of an automatic differentiation algorithm (e.g., including DFS) optimizes the method for generating Shapley values (e.g., by caching frequently used intermediate outputs during the forward path for backpropagation) and simplifies the computational graph for generating Shapley values (e.g., a reverse sign graph based on a forward sign graph).

[0163] As Figure 3C As shown in FIG. 320 , the model interpretation system 102 may receive a real-time inference request for a neural network machine learning model from a user device 106 . Figure 3C As further shown in FIG. 325 , the model interpretation system 102 may determine an output of a neural network machine learning model associated with a real-time inference request.

[0164] In some non-limiting embodiments or aspects, the model interpretation system 102 may use a neural network machine learning model to determine an output of the neural network machine learning model associated with an input included in a real-time inference request. For example, the model interpretation system 102 may generate a score (e.g., a model score, a prediction score, etc.) based on the input provided to the neural network machine learning model. In such examples, the model interpretation system 102 may generate a score based on the input included with the inference request and provided to the neural network machine learning model to generate a score. In some non-limiting embodiments or aspects, the score of an input (e.g., a data instance) may be equal to the average model score (e.g., the average model score of all inputs in the plurality of inputs) added to the sum of the Shapley values of each feature in the plurality of features included in the input.

[0165] As Figure 3D As shown at reference numeral 330 in FIG, the model interpretation system 102 may determine one or more Shapley values associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 may determine one or more Shapley values associated with the output of the neural network machine learning model based on an inverse sign graph of the neural network machine learning model, a plurality of intermediate weights, and / or a plurality of reference outputs (e.g., a plurality of reference outputs of the neural network machine learning model stored in a cache memory location). In some non-limiting embodiments or aspects, when determining the one or more Shapley values associated with the output of the neural network machine learning model, the model interpretation system 102 may apply an automatic differentiation algorithm to the inverse sign graph.

[0166] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., determine) a score associated with an inference task based on an output of a neural network machine learning model, the output of the neural network machine learning model being generated based on input data provided as input to the neural network machine learning model (e.g., input data included in an inference request). In one example, the model interpretation system 102 may generate a fraud detection score based on the output of the neural network machine learning model, and one or more Shapley values associated with the output of the neural network machine learning model may include an indication of one or more features of the input data that affect the fraud detection score.

[0167] Now refer to Figure 4 , a diagram illustrating a non-limiting embodiment or aspect of an exemplary environment 400 in which methods, systems, and / or products as described herein may be implemented. Figure 4 As shown in , environment 400 may include a transaction service provider system 402, an issuer system 404, a client device 406, a merchant system 408, an acquirer system 410, and a communication network 412. In some non-limiting embodiments or aspects, Figure 1 Each of the model interpretation system 102, the ML model management database 104, and / or the user device 106 may be implemented by (e.g., a portion of) the transaction service provider system 402. In some non-limiting embodiments or aspects, Figure 1 At least one of the model interpretation system 102, the ML model management database 104 and / or the user device 106 may be implemented by another system, another device, another group of systems or another group of devices that is separate from or includes the transaction service provider system 402 (e.g., a portion thereof), such as by the issuer system 404, the customer device 406, the merchant system 408, the acquirer system 410, etc.

[0168] The transaction service provider system 402 may include one or more devices capable of receiving information from and / or transmitting information to the issuer system 404, the client device 406, the merchant system 408, and / or the acquirer system 410 via a communication network 412. For example, the transaction service provider system 402 may include a computing device, such as a server (e.g., a transaction processing server), a server cluster, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 402 may be associated with a transaction service provider as described herein. In some non-limiting embodiments or aspects, the transaction service provider system 402 may communicate with a data storage device, which may be local or remote to the transaction service provider system 402. In some non-limiting embodiments or aspects, the transaction service provider system 402 may be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device.

[0169] The issuer system 404 may include one or more devices capable of receiving information and / or transmitting information to the transaction service provider system 402, the client device 406, the merchant system 408, and / or the acquirer system 410 via the communication network 412. For example, the issuer system 404 may include a computing device, such as a server, a server cluster, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 404 may be associated with an issuer organization as described herein. For example, the issuer system 404 may be associated with an issuer organization that issues credit accounts, debit accounts, credit cards, debit cards, etc. to users associated with the client device 406.

[0170] The client device 406 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 402, the issuer system 404, the merchant system 408, and / or the acquirer system 410 via the communication network 412. Additionally or alternatively, each client device 406 may include a device capable of receiving information from and / or transmitting information to other client devices 406 via the communication network 412, another network (e.g., a temporary network, a local network, a private network, a virtual private network, etc.), and / or any other suitable communication technology. For example, the client device 406 may include a client device, etc. In some non-limiting embodiments or aspects, the client device 406 may or may not be capable of communicating via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, communication connection, etc.) (e.g., from merchant system 408 or from another client device 406), and / or transmit information (e.g., to merchant system 408) via a short-range wireless communication connection.

[0171] The merchant system 408 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 402, the issuer system 404, the customer device 406, and / or the acquirer system 410 via the communication network 412. The merchant system 408 may also include a communication connection (e.g., an NFC communication connection, an RFID communication connection, a NFC communication connection, a ... Communication connection, 402. In some non-limiting embodiments or aspects, the merchant system 408 may include a computing device, such as a server, a server group, a client device, a client device group, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 408 may be associated with the merchant described herein. In some non-limiting embodiments or aspects, the merchant system 408 may include one or more client devices. For example, the merchant system 408 may include a client device that allows the merchant to transmit information to the transaction service provider system 402. In some non-limiting embodiments or aspects, the merchant system 408 may include one or more devices that can be used by merchants to conduct transactions with users, such as computers, computer systems, and / or peripheral devices. For example, the merchant system 408 may include a POS device and / or a POS system.

[0172] The acquirer system 410 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 402, the issuer system 404, the client device 406, and / or the merchant system 408 via the communication network 412. For example, the acquirer system 410 may include a computing device, a server, a server group, etc. In some non-limiting embodiments or aspects, the acquirer system 410 may be associated with an acquirer as described herein.

[0173] The communication network 412 may include one or more wired and / or wireless networks. For example, the communication network 412 may include a cellular network (e.g., a long-term evolution (LTE) network, a third-generation (3G) network, a fourth-generation (4G) network, a fifth-generation (5G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network (e.g., a private network associated with a transaction service provider), an ad hoc network, an intranet, the Internet, a fiber-optic-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0174] Figure 4 The number and arrangement of systems, devices, and / or networks shown in are provided as examples. Figure 4 There may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or systems, devices, and / or networks arranged differently than those shown in FIG. Furthermore, a single system and / or device may be implemented Figure 4 Two or more systems or devices shown in, or Figure 4 A single system or device shown in the environment 400 may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of the environment 400 may perform one or more functions described as being performed by another group of systems or another group of devices of the environment 400.

[0175] Now refer to Figure 5 , a diagram showing example components of an apparatus 500 according to a non-limiting embodiment or aspect. As an example, the apparatus 500 may correspond to Figure 1 At least one of the model interpretation system 102, the ML model management database 104, and / or the user device 106 and / or Figure 4In some non-limiting embodiments or aspects, at least one of the transaction service provider system 402, the issuer system 404, the client device 406, the merchant system 408 and / or the acquirer system 410. Figure 1 or Figure 4 Such a system or device in may include at least one device 500 and / or at least one component of device 500. Figure 5 The number and arrangement of components shown in are provided as examples. In some non-limiting embodiments or aspects, Figure 5 , device 500 may include additional components, fewer components, different components, or components arranged in a different manner than those shown in . Additionally or alternatively, one set of components (e.g., one or more components) of device 500 may perform one or more functions described as being performed by another set of components of device 500.

[0176] like Figure 5 As shown in FIG, device 500 may include a bus 502, a processor 504, a memory 506, a storage component 508, an input component 510, an output component 512, and a communication interface 514. Bus 502 may include components that permit communication between components of device 500. In some non-limiting embodiments or aspects, processor 504 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 504 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.). Memory 506 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 504.

[0177] Continue to refer Figure 5, storage component 508 may store information and / or software related to the operation and use of device 500. For example, storage component 508 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 510 may include components that permit device 500 to receive information, for example, through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, a microphone, etc.). Additionally or alternatively, input component 510 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 512 may include components that provide output information from device 500 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 514 may include transceiver-type components (e.g., a transceiver, a separate receiver and transmitter, etc.) that enable device 500 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 514 may allow the device 500 to receive information from another device and / or provide information to another device. For example, the communication interface 514 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, interface, cellular network interface, etc.

[0178] The device 500 can perform one or more processes described herein. The device 500 can perform these processes based on the processor 504 executing software instructions stored by a computer-readable medium such as a memory 506 and / or storage component 508. The computer-readable medium may include any non-transitory memory device. The memory device includes a memory space located within a single physical storage device or a memory space extended across multiple physical storage devices. The software instructions can be read from another computer-readable medium or from another device to the memory 506 and / or storage component 508 via the communication interface 514. When executed, the software instructions stored in the memory 506 and / or storage component 508 can cause the processor 504 to perform one or more processes described herein. In addition or alternatively, hard-wired circuits can replace or be used in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuits and software. As used herein, the term "configured to" can refer to the arrangement of software, devices and / or hardware for performing and / or implementing one or more functions (e.g., actions, processes, steps of processes, etc.). For example, a "processor configured to" may refer to a processor executing software instructions (eg, program code) that cause the processor to perform one or more functions.

[0179] Although the embodiments have been described in detail for purposes of illustration, it should be understood that such detail is solely for that purpose and that the present disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method comprising: receiving, with at least one processor, an executable file for a neural network machine learning model; Converting, using at least one processor, the format of the executable file for the neural network machine learning model into a model-agnostic format to provide the neural network machine learning model with a model-agnostic format file; Parsing, using at least one processor, the model-agnostic format file for the neural network machine learning model, wherein parsing the model-agnostic format file for the neural network machine learning model comprises: storing, with at least one processor, a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating, with at least one processor, a forward sign graph associated with the neural network machine learning model; and generating, with at least one processor, a reverse sign graph associated with the neural network machine learning model based on the forward sign graph; receiving, with at least one processor, a real-time inference request for the neural network machine learning model; determining, with at least one processor, an output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and Determine, with at least one processor, one or more Shapley values associated with the output of the neural network machine learning model based on the inverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

2. The computer-implemented method of claim 1 , further comprising: A loss function is generated for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

3. The computer-implemented method of claim 2 , wherein generating the reverse symbolic graph associated with the neural network machine learning model comprises: The inverse sign graph associated with the neural network machine learning model is generated based on the loss function for the neural network machine learning model.

4. The computer-implemented method of claim 1 , wherein the forward-signed graph comprises a plurality of nodes and a plurality of edges, and wherein generating the reverse-signed graph associated with the neural network machine learning model comprises: calculating gradients between adjacent nodes of the plurality of nodes of the forward signed graph; as well as A plurality of nodes and a plurality of edges of the reverse-sign graph are generated based on the gradient between adjacent nodes of the plurality of nodes of the forward-sign graph.

5. The computer-implemented method of claim 1 , wherein generating the reverse symbolic graph associated with the neural network machine learning model comprises: The reverse symbolic graph associated with the neural network machine learning model is generated to include a linear operator and a nonlinear operator.

6. The computer-implemented method of claim 1 , wherein determining the one or more Shapley values associated with the output of the neural network machine learning model comprises: An automatic differentiation algorithm is applied to the inverse sign graph.

7. The computer-implemented method of claim 1 , further comprising: determining a fraud detection score based on the output of the neural network machine learning model, wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

8. A system comprising: at least one processor configured to: receiving an executable file for a neural network machine learning model; Converting the format of the executable file for the neural network machine learning model into an agnostic model format to provide the neural network machine learning model with an agnostic model format file; parsing the model-agnostic format file for the neural network machine learning model, wherein when parsing the model-agnostic format file for the neural network machine learning model, the at least one processor is configured to: storing a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse signed graph associated with the neural network machine learning model based on the forward signed graph; receiving a real-time inference request for the neural network machine learning model; determining, using the neural network machine learning model, an output of the neural network machine learning model associated with the real-time inference request; and One or more Shapley values associated with the output of the neural network machine learning model are determined based on the inverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

9. The system of claim 8, wherein the at least one processor is further configured to: A loss function is generated for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

10. The system of claim 9, wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: The inverse sign graph associated with the neural network machine learning model is generated based on the loss function for the neural network machine learning model.

11. The system of claim 8, wherein the forward-signed graph comprises a plurality of nodes and a plurality of edges, and wherein when generating the reverse-signed graph associated with the neural network machine learning model, the at least one processor is configured to: calculating gradients between adjacent nodes of the plurality of nodes of the forward sign graph; and A plurality of nodes and a plurality of edges of the reverse-sign graph are generated based on the gradient between adjacent nodes of the plurality of nodes of the forward-sign graph.

12. The system of claim 8, wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: The reverse symbolic graph associated with the neural network machine learning model is generated to include a linear operator and a nonlinear operator.

13. The system of claim 8, wherein when determining the one or more Shapley values associated with the output of the neural network machine learning model, the at least one processor is configured to: An automatic differentiation algorithm is applied to the inverse sign graph.

14. The system of claim 8, wherein the at least one processor is further configured to: determining a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

15. A computer program product comprising at least one non-transitory computer-readable medium containing program instructions that, when executed by at least one processor, cause the at least one processor to: receiving an executable file for a neural network machine learning model; Converting the format of the executable file for the neural network machine learning model into an agnostic model format to provide the neural network machine learning model with an agnostic model format file; parsing the model-agnostic format file for the neural network machine learning model, wherein the program instructions causing the at least one processor to parse the model-agnostic format file for the neural network machine learning model cause the at least one processor to: storing a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward sign graph associated with the neural network machine learning model; and generating a reverse signed graph associated with the neural network machine learning model based on the forward signed graph; receiving a real-time inference request for the neural network machine learning model; determining, using the neural network machine learning model, an output of the neural network machine learning model associated with the real-time inference request; and One or more Shapley values associated with the output of the neural network machine learning model are determined based on the inverse sign graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

16. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: A loss function is generated for the neural network machine learning model based on a difference between an output of the forward sign graph and the reference input data provided to the neural network machine learning model.

17. The computer program product of claim 16, wherein the program instructions that cause the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: The inverse sign graph associated with the neural network machine learning model is generated based on the loss function for the neural network machine learning model.

18. The computer program product of claim 15, wherein the forward sign graph comprises a plurality of nodes and a plurality of edges, and wherein the program instructions that cause the at least one processor to generate the reverse sign graph associated with the neural network machine learning model cause the at least one processor to: calculating gradients between adjacent nodes of the plurality of nodes of the forward sign graph; and A plurality of nodes and a plurality of edges of the reverse-sign graph are generated based on the gradient between adjacent nodes of the plurality of nodes of the forward-sign graph.

19. The computer program product of claim 15, wherein the program instructions that cause the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: The reverse symbolic graph associated with the neural network machine learning model is generated to include a linear operator and a nonlinear operator.

20. The computer program product of claim 15, wherein the program instructions that cause the at least one processor to determine the one or more Shapley values associated with the output of the neural network machine learning model cause the at least one processor to: An automatic differentiation algorithm is applied to the inverse sign graph.

21. The computer program product of claim 15, wherein the program instructions further cause the at least one processor to: determining a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values associated with the output of the neural network machine learning model include an indication of one or more features of the input data included in the real-time inference request that affect the fraud detection score.

Images