Implementing a model-agnostic framework to provide Shapley values associated with machine learning models

Abstract

Methods, systems, and computer program products are provided for implementing a model-agnostic framework to provide Shapley values associated with machine learning models. One method can include receiving an executable file for a neural network machine learning model; converting a format of the executable file for the neural network machine learning model to an agnostic model format to provide an agnostic model format file for the neural network machine learning model; parsing the agnostic model format file to provide a forward symbolic graph associated with the neural network machine learning model and a backward symbolic graph associated with the neural network machine learning model; receiving a real-time inference request; and determining an output of the neural network machine learning model associated with the real-time inference request and one or more Shapley values associated with the output of the neural network machine learning model.

Description

[0001] Cross-referencing related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 526,230, filed July 12, 2023, the disclosure of which is incorporated herein by reference in its entirety. Background Technology 1. Technical Field

[0004] This disclosure generally relates to the analysis of machine learning models, and in some specific embodiments or aspects, to methods, systems, and computer program products for implementing a model agnostic framework to provide Shapley values ​​associated with machine learning models.

[0005] 2. Technical considerations

[0006] Model interpretability (e.g., model interpretability or machine learning model transparency) can refer to the ability to understand a machine learning model. In some instances, model interpretability may include machine learning interpretations, which are a set of views of the model's functions that help users understand the outcomes predicted by the machine learning model. Some methods for providing model interpretations may include logistic regression coefficients, LIME, Shapley value techniques (e.g., QII, SHAP), and integral gradient interpretations.

[0007] Shapley-based techniques can be algorithmically interpretable and / or model-agnostic. They assume no access to the model's internals and can be applied to any model type. Shapley-based techniques may involve core algorithms applicable to any input but useful for interpreting the constituent features of a machine learning model. Other Shapley-based interpretations can be used to determine local and / or global model inference for various model outputs (e.g., probability, regression, classification results, etc.).

[0008] The Shapley value can be defined as a credit assigned to the output of a machine learning model among the input features that produce the output, using a fair allocation outcome from cooperative game theory. In some instances, the Shapley value can be calculated by carefully perturbing the input features and observing how the changes in the input features correspond to the final model prediction. Subsequently, the Shapley value of a given feature can be calculated as the average marginal contribution to the final model prediction (e.g., the overall model score).

[0009] However, current model interpretability techniques, such as SHAP, may not be able to simultaneously obtain both output (e.g., model score) and model explanation. Furthermore, such techniques may require significant resources to provide the explanation and may consume substantial amounts of memory. Additionally, such techniques may only support machine learning models written in specific languages, such as SHAP's requirement for machine learning models written in PyTorch or TensorFlow. Summary of the Invention

[0010] Therefore, methods, systems, and computer program products are provided for implementing a model-agnostic framework to provide improved Shapley values ​​associated with machine learning models.

[0011] According to a non-limiting embodiment or aspect, a method is provided for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model, comprising: receiving an executable file for a neural network machine learning model; converting the format of the executable file for the neural network machine learning model to an model-agnostic format to provide an model-agnostic format file for the neural network machine learning model; parsing the model-agnostic format file for the neural network machine learning model, wherein parsing the model-agnostic format file for the neural network machine learning model includes: storing a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generating a forward symbol graph associated with the neural network machine learning model; and generating a backward symbol graph associated with the neural network machine learning model based on the forward symbol graph; receiving a real-time inference request for the neural network machine learning model; determining the output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determining one or more Shapley values ​​associated with the output of the neural network machine learning model based on the backward symbol graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0012] In some non-limiting embodiments or aspects, the method further includes generating a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0013] In some non-limiting embodiments or aspects, generating an inverse symbol graph associated with a neural network machine learning model includes generating an inverse symbol graph associated with a neural network machine learning model based on a loss function used for the neural network machine learning model.

[0014] In some non-limiting embodiments or aspects, the forward symbolic graph includes a plurality of nodes and a plurality of edges, and the generation of the reverse symbolic graph associated with the neural network machine learning model includes: calculating the gradient between adjacent nodes in the plurality of nodes of the forward symbolic graph; and generating the plurality of nodes and the plurality of edges of the reverse symbolic graph based on the gradient between adjacent nodes in the plurality of nodes of the forward symbolic graph.

[0015] In some non-limiting embodiments or aspects, generating the inverse symbol graph associated with the neural network machine learning model includes: generating the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0016] In some non-limiting embodiments or aspects, determining one or more Shapley values ​​associated with the output of a neural network machine learning model includes applying an automatic differentiation algorithm to the inverse symbol graph.

[0017] In some non-limiting embodiments or aspects, the method further includes: determining a fraud detection score based on the output of a neural network machine learning model, wherein one or more Shapley values ​​associated with the output of the neural network machine learning model include indications of one or more features of the input data that influence the fraud detection score contained in the real-time inference request.

[0018] According to a non-limiting embodiment or aspect, a system is provided for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model, comprising at least one processor configured to: receive an executable file for a neural network machine learning model; convert the format of the executable file for the neural network machine learning model to an model-agnostic format to provide an model-agnostic format file for the neural network machine learning model; parse the model-agnostic format file for the neural network machine learning model, wherein when parsing the model-agnostic format file for the neural network machine learning model, the at least one processor is configured to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; generate a forward symbol graph associated with the neural network machine learning model; and generate a backward symbol graph associated with the neural network machine learning model based on the forward symbol graph; receive a real-time inference request for the neural network machine learning model; determine the output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model; and determine one or more Shapley values ​​associated with the output of the neural network machine learning model based on the backward symbol graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0019] In some non-limiting embodiments or aspects, at least one processor is further configured to generate a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0020] In some non-limiting embodiments or aspects, when generating an inverse symbol graph associated with a neural network machine learning model, at least one processor is configured to generate the inverse symbol graph associated with the neural network machine learning model based on a loss function used for the neural network machine learning model.

[0021] In some non-limiting embodiments or aspects, the forward symbolic graph includes a plurality of nodes and a plurality of edges, and wherein when generating the reverse symbolic graph associated with a neural network machine learning model, at least one processor is configured to: compute gradients between adjacent nodes among the plurality of nodes in the forward symbolic graph; and generate the plurality of nodes and a plurality of edges in the reverse symbolic graph based on the gradients between adjacent nodes among the plurality of nodes in the forward symbolic graph.

[0022] In some non-limiting embodiments or aspects, when generating the inverse symbol graph associated with the neural network machine learning model, at least one processor is configured to generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0023] In some non-limiting embodiments or aspects, when determining one or more Shapley values ​​associated with the output of a neural network machine learning model, at least one processor is configured to apply an automatic differentiation algorithm to the inverse symbol graph.

[0024] In some non-limiting embodiments or aspects, at least one processor is further configured to: determine a fraud detection score based on the output of a neural network machine learning model; and wherein one or more Shapley values ​​associated with the output of the neural network machine learning model include indications of one or more features of the input data that influence the fraud detection score contained in the real-time inference request.

[0025] According to a non-limiting embodiment or aspect, a computer program product is provided for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model, comprising at least one non-transient computer-readable medium containing program instructions that, when executed by at least one processor, cause at least one processor to: receive an executable file for a neural network machine learning model; convert the format of the executable file for the neural network machine learning model to an agnostic model format to provide an agnostic model format file for the neural network machine learning model; parse the agnostic model format file for the neural network machine learning model, wherein the program instructions causing at least one processor to parse the agnostic model format file for the neural network machine learning model cause at least one processor to: The neural machine learning model stores multiple intermediate weights and multiple reference outputs in a cache memory location, wherein the multiple intermediate weights and multiple reference outputs of the neural machine learning model are based on reference input data provided to the neural machine learning model; generates a forward symbol map associated with the neural machine learning model; and generates a backward symbol map associated with the neural machine learning model based on the forward symbol map; receives a real-time inference request for the neural machine learning model; uses the neural machine learning model to determine the output of the neural machine learning model associated with the real-time inference request; and determines one or more Shapley values ​​associated with the output of the neural machine learning model based on the backward symbol map and the multiple intermediate weights and multiple reference outputs of the neural machine learning model stored in the cache memory location.

[0026] In some non-limiting embodiments or aspects, the program instructions further cause at least one processor to generate a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0027] In some non-limiting embodiments or aspects, program instructions that cause at least one processor to generate an inverse symbolic graph associated with a neural network machine learning model cause at least one processor to generate an inverse symbolic graph associated with a neural network machine learning model based on a loss function used for the neural network machine learning model.

[0028] In some non-limiting embodiments or aspects, the forward symbol graph includes a plurality of nodes and a plurality of edges, and wherein program instructions that cause at least one processor to generate a reverse symbol graph associated with a neural network machine learning model cause at least one processor to: compute gradients between adjacent nodes in the plurality of nodes of the forward symbol graph; and generate the plurality of nodes and the plurality of edges of the reverse symbol graph based on the gradients between adjacent nodes in the plurality of nodes of the forward symbol graph.

[0029] In some non-limiting embodiments or aspects, program instructions that cause at least one processor to generate an inverse symbol graph associated with a neural network machine learning model cause at least one processor to: generate an inverse symbol graph associated with a neural network machine learning model to include a linear operator and a nonlinear operator.

[0030] In some non-limiting embodiments or aspects, program instructions that cause at least one processor to determine one or more Shapley values ​​associated with the output of a neural network machine learning model cause at least one processor to: apply an automatic differentiation algorithm to the inverse symbol graph.

[0031] In some non-limiting embodiments or aspects, the program instructions further cause at least one processor to: determine a fraud detection score based on the output of a neural network machine learning model; and wherein one or more Shapley values ​​associated with the output of the neural network machine learning model include indications of one or more features of the input data that affect the fraud detection score contained in the real-time inference request.

[0032] Other non-limiting embodiments or aspects will be set forth in the following numbered clauses:

[0033] Clause 1: A computer-implemented method comprising: receiving an executable file for a neural network machine learning model using at least one processor; converting the format of the executable file for the neural network machine learning model into an unknown model format using at least one processor to provide an unknown model format file for the neural network machine learning model; and parsing the unknown model format file for the neural network machine learning model using at least one processor, wherein parsing the unknown model format file for the neural network machine learning model comprises: storing a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location using at least one processor, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on data provided to the neural network machine learning model. The network machine learning model includes: reference input data; generating a forward symbolic graph associated with the neural network machine learning model using at least one processor; generating a backward symbolic graph associated with the neural network machine learning model using at least one processor based on the forward symbolic graph; receiving a real-time inference request for the neural network machine learning model using at least one processor; determining the output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model using at least one processor; and determining one or more Shapley values ​​associated with the output of the neural network machine learning model based on the backward symbolic graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location using at least one processor.

[0034] Clause 2: The computer-implemented method according to Clause 1 further comprises: generating a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0035] Clause 3: A computer-implemented method according to Clause 1 or 2, wherein generating the inverse symbol graph associated with the neural network machine learning model comprises: generating the inverse symbol graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0036] Clause 4: A computer-implemented method according to any one of Clauses 1 to 3, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein generating the reverse symbolic graph associated with the neural network machine learning model comprises: calculating gradients between adjacent nodes among the plurality of nodes of the forward symbolic graph; and generating the plurality of nodes and a plurality of edges of the reverse symbolic graph based on the gradients between adjacent nodes among the plurality of nodes of the forward symbolic graph.

[0037] Clause 5: A computer-implemented method according to any one of Clauses 1 to 4, wherein generating the inverse symbol graph associated with the neural network machine learning model comprises: generating the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0038] Clause 6: A computer-implemented method according to any one of Clauses 1 to 5, wherein determining the one or more Shapley values ​​associated with the output of the neural network machine learning model comprises: applying an automatic differentiation algorithm to the inverse symbol graph.

[0039] Clause 7: A computer-implemented method according to any one of Clauses 1 to 6, further comprising: determining a fraud detection score based on the output of the neural network machine learning model, wherein the one or more Shapley values ​​associated with the output of the neural network machine learning model include an indication of one or more features of the input data that affect the fraud detection score contained in the real-time inference request.

[0040] Clause 8: A system comprising: at least one processor configured to: receive an executable file for a neural network machine learning model; convert the format of the executable file for the neural network machine learning model into an agnostic model format to provide an agnostic model format file for the neural network machine learning model; parse the agnostic model format file for the neural network machine learning model, wherein when parsing the agnostic model format file for the neural network machine learning model, the at least one processor is configured to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache memory location, wherein the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model... Multiple reference outputs are provided based on reference input data to the neural network machine learning model; a forward symbol map associated with the neural network machine learning model is generated; and a backward symbol map associated with the neural network machine learning model is generated based on the forward symbol map; a real-time inference request for the neural network machine learning model is received; the output of the neural network machine learning model associated with the real-time inference request is determined using the neural network machine learning model; and one or more Shapley values ​​associated with the output of the neural network machine learning model are determined based on the backward symbol map, the multiple intermediate weights of the neural network machine learning model stored in the cache memory location, and the multiple reference outputs.

[0041] Clause 9: The system according to Clause 8, wherein the at least one processor is further configured to generate a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0042] Clause 10: A system according to Clause 8 or 9, wherein when generating the inverse symbol graph associated with the neural network machine learning model, the at least one processor is configured to generate the inverse symbol graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0043] Clause 11: A system according to any one of Clauses 8 to 10, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: compute gradients between adjacent nodes among the plurality of nodes of the forward symbolic graph; and generate the plurality of nodes and the plurality of edges of the reverse symbolic graph based on the gradients between adjacent nodes among the plurality of nodes of the forward symbolic graph.

[0044] Clause 12: A system according to any one of Clauses 8 to 11, wherein when generating the inverse symbol graph associated with the neural network machine learning model, the at least one processor is configured to: generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0045] Clause 13: A system according to any one of Clauses 8 to 12, wherein when determining the one or more Shapley values ​​associated with the output of the neural network machine learning model, the at least one processor is configured to apply an automatic differentiation algorithm to the inverse symbol graph.

[0046] Clause 14: A system according to any one of Clauses 8 to 13, wherein the at least one processor is further configured to: determine a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values ​​associated with the output of the neural network machine learning model include an indication of one or more features of the input data that affect the fraud detection score contained in the real-time inference request.

[0047] Clause 15: A computer program product comprising at least one non-transient computer-readable medium containing program instructions that, when executed by at least one processor, cause the at least one processor to: receive an executable file for a neural network machine learning model; convert the format of the executable file for the neural network machine learning model into an agnostic model format to provide an agnostic model format file for the neural network machine learning model; parse the agnostic model format file for the neural network machine learning model, wherein the program instructions causing the at least one processor to parse the agnostic model format file for the neural network machine learning model cause the at least one processor to: store a plurality of intermediate weights and a plurality of reference outputs of the neural network machine learning model in a cache. In a cache memory location, the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model; a forward symbol map associated with the neural network machine learning model is generated; and a reverse symbol map associated with the neural network machine learning model is generated based on the forward symbol map; a real-time inference request for the neural network machine learning model is received; the neural network machine learning model is used to determine the output of the neural network machine learning model associated with the real-time inference request; and one or more Shapley values ​​associated with the output of the neural network machine learning model are determined based on the reverse symbol map and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location.

[0048] Clause 16: A computer program product according to Clause 15, wherein the program instructions further cause the at least one processor to generate a loss function for the neural network machine learning model based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

[0049] Clause 17: A computer program product pursuant to Clause 15 or 16, wherein the program instructions that cause the at least one processor to generate the inverse symbol graph associated with the neural network machine learning model cause the at least one processor to generate the inverse symbol graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0050] Clause 18: A computer program product according to any one of Clauses 15 to 17, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein the program instructions that cause the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: calculate the gradient between adjacent nodes among the plurality of nodes of the forward symbolic graph; and generate the plurality of nodes and the plurality of edges of the reverse symbolic graph based on the gradient between adjacent nodes among the plurality of nodes of the forward symbolic graph.

[0051] Clause 19: A computer program product according to any one of Clauses 15 to 18, wherein the program instructions that cause the at least one processor to generate the inverse symbol graph associated with the neural network machine learning model cause the at least one processor to: generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

[0052] Clause 20: A computer program product according to any one of Clauses 15 to 19, wherein the program instructions that cause the at least one processor to determine the one or more Shapley values ​​associated with the output of the neural network machine learning model cause the at least one processor to: apply an automatic differentiation algorithm to the inverse symbol graph.

[0053] Clause 21: A computer program product according to any one of Clauses 15 to 20, wherein the program instructions further cause the at least one processor to: determine a fraud detection score based on the output of the neural network machine learning model; and wherein the one or more Shapley values ​​associated with the output of the neural network machine learning model include an indication of one or more features of the input data that affect the fraud detection score contained in the real-time inference request.

[0054] These and other features and characteristics of this disclosure, as well as the operational methods and manufacturing economies of combinations of related structural elements and parts, will become more apparent when considered in conjunction with the accompanying drawings, all of which form part of this specification, wherein similar reference numerals denote corresponding parts in the figures. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be a definition of limitation on the disclosed subject matter. Attached Figure Description

[0055] Additional advantages and details of this disclosure are explained in more detail below with reference to exemplary embodiments illustrated in the accompanying drawings, in which:

[0056] Figure 1The figures are non-limiting embodiments or aspects of environments in which the systems, apparatuses, products, devices and / or methods described herein can be implemented based on the principles of this disclosure;

[0057] Figure 2 This is a flowchart of a non-limiting embodiment or aspect of a process for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model;

[0058] Figures 3A-3D This is a schematic diagram of an exemplary implementation of a system and / or method for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model, based on some non-limiting embodiments or aspects.

[0059] Figure 4 These are diagrams illustrating exemplary environments in which the systems, methods, and / or computer program products described herein may be implemented, based on some non-limiting embodiments or aspects; and

[0060] Figure 5 Based on some non-limiting embodiments or aspects Figure 1 and / or Figure 4 A schematic diagram of example components of one or more devices. Detailed Implementation

[0061] For the purposes of the following description, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives should be associated with the orientation of the embodiments in the accompanying drawings. However, it should be understood that embodiments may employ various alternative variations and sequences of steps, except where explicitly specified as the opposite. It should also be understood that the specific apparatus and processes shown in the drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered limiting.

[0062] This document may describe some non-limiting embodiments or aspects in conjunction with thresholds. As used herein, a threshold may refer to a value greater than a threshold, more than a threshold, higher than a threshold, greater than or equal to a threshold, less than a threshold, less than a threshold, lower than a threshold, less than or equal to a threshold, equal to a threshold, etc.

[0063] The aspects, components, elements, structures, actions, steps, functions, instructions, etc., used herein should not be construed as critical or essential unless explicitly described as such. Furthermore, as used herein, the article “a” is intended to include one or more items and is interchangeable with “one or more” and “at least one.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with “one or more” or “at least one.” The term “a” or similar language is used where only one item is desired. Moreover, as used herein, the term “having” and the like are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, a reference to an action “based on” a condition may mean that the action is “in response to” the condition. For example, in some non-limiting embodiments or aspects, the phrases “based on” and “in response to” may refer to conditions that automatically trigger an action (e.g., a specific operation of an electronic device, such as a computing device, processor, etc.).

[0064] As used herein, the term "acquiring institution" can refer to an entity licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using payment devices associated with the transaction service provider. Transactions that an acquiring institution can initiate may include payment transactions (e.g., purchases, original letter of credit transactions (OCT), account treasury transactions (AFT), etc.). In some non-limiting embodiments or aspects, the acquiring institution may be a financial institution, such as a bank. As used herein, the term "acquiring system" can refer to one or more computing devices operated by or on behalf of an acquiring institution, such as a server computer executing one or more software applications.

[0065] As used herein, the term "account identifier" may include one or more master account (PAN), tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a substitute or replacement identifier for the original account identifier, such as a PAN. Account identifiers may be any combination of alphanumeric or characters and / or symbols. Tokens may be associated with PANs or other original account identifiers in one or more data structures (e.g., one or more databases, etc.) such that the tokens can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier, such as a PAN, may be associated with multiple tokens for different individuals or purposes.

[0066] As used herein, the term "communication" can refer to the receiving, accepting, transmitting, conveying, providing, etc., of data (e.g., information, signals, messages, instructions, commands, etc.). For one unit (e.g., apparatus, system, component of an apparatus or system, combination thereof, etc.) to communicate with another unit means that the first unit is able to receive information directly or indirectly from and / or transmit information to the other unit. This can refer to a direct or indirect connection that is inherently wired and / or wireless (e.g., a direct communication connection, an indirect communication connection, etc.). Furthermore, although the transmitted information may be modified, processed, relayed, and / or routed between the first and second units, the two units can also communicate with each other. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively transmit information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message can refer to a network packet (e.g., a data packet, etc.) containing data. It should be understood that many other arrangements are possible.

[0067] As used herein, the term "computing device" can refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, display, memory, input device, network interface, etc. A computing device may be a mobile device. As examples, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other forms of non-mobile computer.

[0068] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communication and processing among multiple parties in a network environment such as the Internet, but it should be understood that communication may be facilitated through one or more public or private network environments, and various other arrangements may be possible. Furthermore, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) communicating directly or indirectly in a network environment may constitute a "system".

[0069] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processor, server, client device, software application, components of such devices, etc.). As used herein, references to "device," "server," "processor," etc., may refer to a previously described device, server, or processor, different devices, servers, or processors, and / or combinations of devices, servers, and / or processors, described as performing a preceding step or function. For example, as used in the specification and claims, a first device, first server, or first processor described as performing a first step or a first function may refer to the same or different devices, servers, or processors described as performing a second step or a second function.

[0070] As used herein, the term "issuer institution" can refer to one or more entities, such as a bank, that provide customers with accounts for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide a customer with an account identifier, such as a PAN, that uniquely identifies one or more accounts associated with said customer. The account identifier may be embodied in a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0071] As used herein, the term "merchant" can refer to an individual or entity that provides goods and / or services or access to goods and / or services to a customer based on a transaction such as a payment transaction. The terms "merchant" or "merchant system" can also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0072] As used herein, the term "payment device" can refer to electronic payment devices, portable financial devices (e.g., payment cards, such as credit or debit cards), gift cards, smart cards, smart media, pay cards, healthcare cards, wristbands, machine-readable media containing account information, keychain devices or pendants, radio frequency identification (RFID) transponders, retailer discount or membership cards, cellular phones, e-wallet mobile applications, PDAs, pagers, security cards, computing devices, access control cards, wireless terminals, transponders, etc. In some non-limiting embodiments or aspects, the payment device may include volatile or non-volatile memory for storing information (e.g., account identifiers, account holder names, etc.).

[0073] As used herein, "point-of-sale (POS) device" can refer to one or more devices that a merchant can use to conduct transactions (e.g., payment transactions) and / or process transactions. For example, a POS device may include one or more client devices. Alternatively or additionally, a POS device may include peripheral devices, card readers, scanning devices (e.g., barcode scanners), Communication receivers, near-field communication (NFC) receivers, RFID receivers and / or other contactless transceivers or receivers, contact-based receivers, payment terminals, etc. As used herein, a "point-of-sale (POS) system" can refer to one or more client devices and / or peripheral devices used by a merchant to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers configured to process online payment transactions via web pages, mobile applications, etc.

[0074] As used herein, the term "transaction service provider" can refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment guarantees through an agreement between the transaction service provider and the issuing entity. For example, a transaction service provider may include, for instance... Payment networks, or any other entity that processes transactions. The term "transaction processing system" can refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may contain one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0075] Non-limiting embodiments or aspects of the disclosed subject matter are methods, systems, and computer program products for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model. In some non-limiting embodiments or aspects, the model interpretation system may receive a file (e.g., an executable file) for a neural network machine learning model, convert the format of the file for the neural network machine learning model to a model-agnostic format (e.g., Open Neural Network Exchange (ONNX) format) to provide a model-agnostic file (e.g., an ONNX file) for the neural network machine learning model, and parse the model-agnostic file for the neural network machine learning model. In some non-limiting embodiments or aspects, when parsing the model-agnostic file for the neural network machine learning model, the model interpretation system may store intermediate weights and multiple reference outputs of the neural network machine learning model in a cache memory location, wherein the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model, generating a forward symbol graph associated with the neural network machine learning model, and generating a backward symbol graph associated with the neural network machine learning model based on the forward symbol graph. In some non-limiting embodiments or aspects, the model interpretation system may receive a real-time inference request for a neural network machine learning model, use the neural network machine learning model to determine the output of the neural network machine learning model associated with the real-time inference request, and determine one or more Shapley values ​​associated with the output of the neural network machine learning model based on a reverse symbolic graph and intermediate weights and multiple reference outputs of the neural network machine learning model stored in a cache memory location.

[0076] In some non-limiting embodiments or aspects, the model interpretation system may generate a loss function for the neural network machine learning model based on the difference between the output of the forward symbolic graph and the reference input data provided to the neural network machine learning model. In some non-limiting embodiments or aspects, when generating the backward symbolic graph associated with the neural network machine learning model, the model interpretation system may generate the backward symbolic graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0077] In some non-limiting embodiments or aspects, the forward symbol graph includes multiple nodes and multiple edges, and when generating the reverse symbol graph associated with a neural network machine learning model, the model interpretation system can compute the gradient between adjacent nodes in the multiple nodes of the forward symbol graph, and generate the multiple nodes and multiple edges of the reverse symbol graph based on the gradient between adjacent nodes in the multiple nodes of the forward symbol graph.

[0078] In some non-limiting embodiments or aspects, when generating a reverse symbol graph associated with a neural network machine learning model, the model interpretation system may generate a reverse symbol graph associated with the neural network machine learning model to include a linear operator and / or a nonlinear operator.

[0079] In some non-limiting embodiments or aspects, the model interpretation system may apply an automatic differentiation algorithm to the inverse symbol graph. In some non-limiting embodiments or aspects, the model interpretation system may determine a fraud detection score based on the output of a neural network machine learning model, and one or more Shapley values ​​associated with the output of the neural network machine learning model include indications of one or more features of the input data that influence the fraud detection score included in the real-time inference request.

[0080] In this way, the model interpretation system can provide outputs (e.g., model scores indicating the accuracy of the machine learning model's inferences) and model interpretations simultaneously (e.g., nearly simultaneously) and in real-time (e.g., at or near the time when the model interpretation system is operating). Furthermore, the model interpretation system can reduce the amount of resources required to provide an interpretation and offer faster response times, while providing a framework independent of the type of framework (e.g., language type) used to initially prepare the machine learning model.

[0081] For illustrative purposes, while the subject matter disclosed herein is described in relation to methods, systems, and computer program products for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model, which provide an interpretation of the machine learning model's output by attributing the contribution of each feature to the final output (e.g., prediction, model score, etc.) to gain insight into the features that influence the machine learning model's output and to aid in understanding and interpreting the behavior of the machine learning model, those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings, such as prediction, regression, classification, fraud prevention, authorization, authentication, feature selection, etc.

[0082] For illustrative purposes, while the subject matter disclosed herein relates to methods, systems, and computer program products for large-scale graph transformer machine learning model network architectures that can be used in association with providing recommendations, those skilled in the art will recognize that the disclosed subject matter is not limited to the non-limiting embodiments or aspects disclosed herein. For example, the methods, systems, and computer program products described herein can be used with a wide variety of settings and / or for determinations (e.g., prediction, classification, regression, etc.), such as for fraud detection / prevention, authorization, authentication, identification, feature selection, payment processing, etc.

[0083] Now for reference Figure 1 , Figure 1 This is a diagram of an example system 100 in which the apparatuses, systems, and / or methods described herein may be implemented. For example... Figure 1 As shown, system 100 includes a model interpretation system 102, a machine learning (ML) model management database 104, a user device 106, and a communication network 108. The model interpretation system 102, the ML model management database 104, and / or the user device 106 may be interconnected (e.g., establishing a connection for communication) via wired connection, wireless connection, or a combination of wired and wireless connection.

[0084] Model interpretation system 102 may include one or more devices capable of receiving information from and / or transmitting information to ML model management database 104 and / or user device 106 via communication network 108 (e.g., directly via wired or wireless communication connection, indirectly via communication network 108, etc.). For example, model interpretation system 102 may include servers, server clusters, cloud platforms, and / or other similar devices. In some non-limiting embodiments or aspects, model interpretation system 102 may be associated with a transaction service provider system. For example, model interpretation system 102 may be operated by a transaction service provider system. In another example, model interpretation system 102 may be a component of user device 106. In another example, model interpretation system 102 may include ML model management database 104. In some non-limiting embodiments or aspects, model interpretation system 102 may communicate with a data storage device (e.g., ML model management database 104), which may be local or remote to model interpretation system 102. In some non-limiting embodiments or aspects, the model interpretation system 102 may be able to receive information from a data storage device, store information in a data storage device, transmit information to a data storage device, and / or search for information stored in a data storage device.

[0085] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., train, validate, retrain, etc.), store, and / or implement (e.g., operate, provide inputs to one or more machine learning models, and / or outputs from one or more machine learning models, etc.) one or more machine learning models. For example, the model interpretation system 102 may generate one or more machine learning models by fitting (e.g., validating, testing, etc.) one or more machine learning models against data used for training (e.g., training data). In some non-limiting embodiments or aspects, the model interpretation system 102 may generate, store, and / or implement one or more machine learning models that are provided to a production environment (e.g., runtime environment, real-time environment, etc.) for providing inference (e.g., secure inference) based on data input in a real-world scenario (e.g., real-time scenario). Alternatively or additionally, the model interpretation system 102 may generate, store, and / or implement one or more machine learning models that are provided to a non-production environment (e.g., offline environment, training environment, etc.) for providing inference based on data input in a non-real-world scenario. In some non-limiting embodiments or aspects, the model interpretation system 102 may communicate with a data storage device (ML model management database 104), which may be local or remote for the model interpretation system 102.

[0086] The ML model management database 104 may include one or more means capable of receiving information from and / or transmitting information to the model interpretation system 102 and / or user device 106, for example, directly via a wired or wireless communication connection, indirectly via a communication network 108, etc. For example, the ML model management database 104 may include a server, server cluster, desktop computer, portable computer, mobile device, and / or other similar means. In some non-limiting embodiments or aspects, the ML model management database 104 may include a data storage device. In some non-limiting embodiments or aspects, the ML model management database 104 may be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device. In some non-limiting embodiments or aspects, the ML model management database 104 may be part of the model interpretation system 102 and / or part of a system identical to the model interpretation system 102.

[0087] User device 106 may include one or more means capable of receiving information from and / or transmitting information to model interpretation system 102 and / or ML model management database 104, such as directly via a wired or wireless communication connection or indirectly via communication network 108. For example, user device 106 may include computing devices, such as mobile devices, portable computers, desktop computers, and / or other similar devices. Alternatively, user device 106 may include means capable of receiving information from and / or transmitting information to other user devices, such as directly via a wired or wireless communication connection or indirectly via communication network 108. In some non-limiting embodiments or aspects, user device 106 may be part of model interpretation system 102 and / or part of the same system as model interpretation system 102. For example, model interpretation system 102, ML model management database 104, and user device 106 may all be a single system and / or a single computing device (and / or a portion thereof).

[0088] The communication network 108 may include one or more wired and / or wireless networks. For example, the communication network 108 may include cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMN), Local Area Networks (LAN), Wide Area Networks (WAN), Metropolitan Area Networks (MAN), Telephone Networks (e.g., Public Switched Telephone Network (PSTN), etc.), Private Networks, Ad Hoc Networks, Intranets, the Internet, Fiber-based Networks, Cloud Computing Networks, etc., and / or combinations of some or all of these or other types of networks.

[0089] Figure 1 The number and arrangement of systems and devices shown are provided as examples. Figure 1 Compared to the systems and / or devices shown, there may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner. Furthermore, implementation may be within a single system and / or device. Figure 1 The two or more systems or devices shown, or Figure 1 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively or additionally, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0090] Now for reference Figure 2A flowchart of process 200 for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model is shown, according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed (e.g., wholly, partially, etc.) by model interpretation system 102 (e.g., one or more means of model interpretation system 102). In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed (e.g., wholly, partially, etc.) by another means or set of means separate from or including model interpretation system 102 (e.g., one or more means of model interpretation system 102), ML model management database 104, and / or user device 106. Figure 2 The steps shown are for illustrative purposes only. It will be understood that additional, fewer, different, and / or different orders of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, steps may be performed automatically in response to the execution and / or completion of previous steps.

[0091] like Figure 2 As shown, at step 202, process 200 includes receiving a file for a machine learning model. For example, model interpretation system 102 may receive a file for a machine learning model. In one example, the file may contain an executable file for a machine learning model, such as a neural network machine learning model. In some non-limiting embodiments or aspects, the file for the machine learning model may have a format based on the type of machine learning framework used to develop the machine learning model (e.g., Keras, PyTorch, TensorFlow, Caffe, Matlab, etc.). In some non-limiting embodiments or aspects, model interpretation system 102 may receive data associated with the machine learning model, which may contain the file for the machine learning model. In some non-limiting embodiments or aspects, model interpretation system 102 may receive data from ML model management database 104, user device 106, and / or another system or device.

[0092] like Figure 2As shown, at step 204, process 200 includes converting the format of the file used for the machine learning model to provide an unknown model format file for the machine learning model. For example, model interpretation system 102 may convert the format of the file used for the machine learning model to an unknown model format to provide an unknown model format file for the machine learning model. In some non-limiting embodiments or aspects, model interpretation system 102 may convert the format of the executable file used for the machine learning model (e.g., a neural network machine learning model) to ONNX format to provide an ONNX file for the machine learning model. Alternatively or additionally, model interpretation system 102 may convert the format of the file used for the machine learning model to a standardized model format (e.g., Predictive Model Markup Language (PMML) format, Portable Analysis Format (PFA) format, TensorFlow SavedModel format, Keras HDF5 format, Core ML format, MXNet model format, Caffe model format, etc.) to provide a standardized model file to be used as an unknown model format file for the machine learning model.

[0093] like Figure 2 As shown, at step 206, process 200 includes parsing an agnostic model format file for a machine learning model to provide a symbolic graph associated with the machine learning model. For example, model interpretation system 102 may parse an agnostic model format file (e.g., an ONNX file) for a machine learning model to provide forward and / or backward symbolic graphs associated with the machine learning model. In some non-limiting embodiments or aspects, the symbolic graph associated with the machine learning model may contain a high-level representation of the computational flow of the machine learning model. The symbolic graph may contain multiple nodes and multiple edges to define the structure (e.g., architecture) and / or operations of the machine learning model. In some non-limiting embodiments or aspects, each node in the symbolic graph may represent an operation (e.g., addition, multiplication, convolution, etc.), and each edge may represent a data flow between operations.

[0094] In some non-limiting embodiments or aspects, the forward symbolic graph associated with a machine learning model may comprise a type of computation graph representing a series of operations required to compute the output of the machine learning model from the inputs provided to the machine learning model. The forward symbolic graph may define the data flow through the machine learning model during forward propagation, where input data is processed to produce outputs (e.g., predictions, model scores, etc.). In some non-limiting embodiments or aspects, nodes in the forward symbolic graph may represent operations and / or layers in the machine learning model and may include mathematical functions, activation functions, layers (e.g., convolutional layers, fully connected layers, etc.) and / or other processing steps. In some non-limiting embodiments or aspects, edges in the forward symbolic graph may represent data flows between nodes, and each edge may represent an output of input passed from one node to another. In some non-limiting embodiments or aspects, the forward symbolic graph may represent the start of one or more input nodes of the raw data provided to the machine learning model, and the forward symbolic graph may represent the end of one or more output nodes of the output. In some non-limiting embodiments or aspects, the forward symbolic graph may be deterministic, such that given the same input, the forward symbolic graph will produce the same output.

[0095] In some non-limiting embodiments or aspects, the backpropagation symbol graph associated with the machine learning model may represent a series of operations required to compute the gradients of the model parameters during backpropagation. In some non-limiting embodiments or aspects, the backpropagation symbol graph may define how gradients propagate back through the machine learning model to update weights. In some non-limiting embodiments or aspects, nodes in the backpropagation symbol graph may represent gradient computations for each operation in forward propagation. Nodes may contain loss function gradients, intermediate activation gradients, and / or model parameter gradients. In some non-limiting embodiments or aspects, edges in the backpropagation symbol graph may represent gradient flows between nodes. Each edge may represent a gradient from one node to a previous node that contributes to the computation of the gradient. In some non-limiting embodiments or aspects, the backpropagation symbol graph may show a flow in the opposite direction to the forward symbol graph. The backpropagation symbol graph may start from a node with a loss and propagate gradients back to one or more input nodes. In some non-limiting embodiments or aspects, each node in the backpropagation symbol graph may correspond to a partial derivative of the loss with respect to one or more variables involved in forward propagation, and the partial derivatives may be used by the model interpretation system 102 to update the model parameters of the machine learning model.

[0096] In some non-limiting embodiments or aspects, the forward symbol graph and / or the reverse symbol graph may contain multiple nodes (e.g., vertices) and multiple edges. In some non-limiting embodiments or aspects, the forward symbol graph and / or the reverse symbol graph may contain a set of nodes (e.g., a set of at least 5, 10, 15, 30, 50, 100, 200, 300, etc., or more nodes) and / or a set of edges (e.g., a set of at least 5, 10, 15, 30, 50, 100, 200, 300, etc., or more edges).

[0097] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate multiple intermediate weights and / or multiple reference outputs of a machine learning model (e.g., a neural network machine learning model) based on reference input data provided to the machine learning model. For example, the model interpretation system 102 may provide reference input data as input to the machine learning model, and the machine learning model may provide multiple reference outputs of the machine learning model based on said input. Multiple intermediate weights may be generated during backpropagation when the model parameters of the machine learning model are updated based on forward propagation of the reference input data.

[0098] In some non-limiting embodiments or aspects, the model interpretation system 102 may receive a dataset containing reference input data (e.g., a training dataset, a reference dataset, etc.). For example, the model interpretation system 102 may receive a dataset from an ML model management database 104. In some non-limiting embodiments or aspects, the reference input data may be associated with one or more entities in an entity group (e.g., users, account holders, merchants, issuers, items provided by entities, etc.). In some non-limiting embodiments or aspects, the reference input data may contain multiple data instances associated with multiple features. In some non-limiting embodiments or aspects, multiple data instances of graph data may represent multiple interactions (e.g., transactions, such as electronic payment transactions) involving one or more entities in a group. In some examples, the reference input data may contain a large number of data instances, such as 100 data instances, 500 data instances, 1,000 data instances, 5,000 data instances, 10,000 data instances, 25,000 data instances, 50,000 data instances, 100,000 data instances, 1,000,000 data instances, etc.

[0099] In some non-limiting embodiments or aspects, each data instance may include transaction data associated with a transaction. In some non-limiting embodiments or aspects, the transaction data may include multiple transaction parameters associated with an electronic payment transaction. In some non-limiting embodiments or aspects, multiple features may represent multiple transaction parameters. In some non-limiting embodiments or aspects, multiple transaction parameters may include e-wallet card data associated with an e-card (e.g., e-credit card, e-debit card, e-membership card, etc.), decision data associated with a decision (e.g., a decision to approve or reject a transaction authorization request), authorization data associated with an authorization response (e.g., approved spending limit, approved transaction value, etc.), PAN, authorization code (e.g., personal identification number (PIN), etc.), data associated with the transaction amount (e.g., approved limit, transaction value, etc.), data associated with the transaction date and time, data associated with the currency exchange rate, data associated with the merchant type (e.g., a merchant category code indicating the type of goods such as groceries, fuel, etc.), data associated with the acquiring institution's country, data associated with an identifier of the country associated with the PAN, data associated with the response code, data associated with the merchant identifier (e.g., merchant name, merchant location, etc.), and data associated with the currency type corresponding to the funds stored in association with the PAN, etc.

[0100] In some non-limiting embodiments or aspects, the model interpretation system 102 may store intermediate weights and / or multiple reference outputs of the machine learning model. For example, the model interpretation system 102 may store the intermediate weights and / or multiple reference outputs of the machine learning model in a cache memory location (e.g., the cache memory location of the model interpretation system 102). In this way, compared to the case where the intermediate weights and / or multiple reference outputs of the machine learning model are stored in another location, the model interpretation system 102 may be able to access the intermediate weights and / or multiple reference outputs of the machine learning model stored in the cache memory location more quickly.

[0101] In some non-limiting embodiments or aspects, the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model. In the above example, the model interpretation system 102 can generate a forward symbolic graph and a backward symbolic graph associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 can generate a backward symbolic graph based on the forward symbolic graph.

[0102] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a loss function for the machine learning model based on the difference between the output of the forward symbolic graph and the reference input data provided to the machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse symbolic graph associated with the neural network machine learning model based on the loss function used for the machine learning model.

[0103] In some non-limiting embodiments or aspects, the model interpretation system 102 may compute gradients associated with the forward symbolic graph. For example, the model interpretation system 102 may compute gradients between adjacent nodes among a plurality of nodes in the forward symbolic graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges in the backward symbolic graph based on the gradients associated with the forward symbolic graph. For example, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges in the backward symbolic graph based on the gradients between adjacent nodes among a plurality of nodes in the forward symbolic graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a backward symbolic graph associated with a neural network machine learning model to include a linear operator and / or a non-linear operator.

[0104] In some non-limiting embodiments or aspects, the model interpretation system 102 may apply an automatic differentiation algorithm to the inverse sign graph. For example, the model interpretation system 102 may apply an automatic differentiation algorithm to the inverse sign graph to optimize (e.g., simplify) the inverse sign graph, which can then be used to generate one or more Shapley values.

[0105] like Figure 2 As shown, at step 208, process 200 includes receiving a real-time inference request for a machine learning model. For example, model interpretation system 102 may receive a real-time inference request for a machine learning model. In some non-limiting embodiments or aspects, the real-time inference request may be based on a task of the machine learning model (e.g., a classification task). For example, the real-time inference request may be based on a request to determine whether a transaction (e.g., a transaction involving a user of user device 106) is fraudulent.

[0106] like Figure 2 As shown, at step 210, process 200 includes determining the output of the machine learning model associated with the real-time inference request and one or more Shapley values ​​associated with the output. For example, model interpretation system 102 may determine the output of the machine learning model associated with the real-time inference request and / or one or more Shapley values ​​associated with the output.

[0107] In some non-limiting embodiments or aspects, the model interpretation system 102 may use a machine learning model to determine the output of the machine learning model associated with the input included in the real-time inference request. For example, the model interpretation system 102 may generate a score (e.g., model score, prediction score, etc.) based on the input provided to the machine learning model. In such examples, the model interpretation system 102 may generate a score based on the input provided to the machine learning model to generate a score, which is included with the inference request. In some non-limiting embodiments or aspects, the score of the input (e.g., a data instance) may be equal to the average model score (e.g., the average model score of all inputs among a plurality of features included in the input) of the sum of the Shapley values ​​of each feature.

[0108] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., determine) a score associated with an inference task based on the output of a machine learning model, the output of which is based on input data (e.g., input data contained in an inference request) provided as input to the machine learning model. In one example, the model interpretation system 102 may generate a fraud detection score based on the output of a machine learning model, and one or more Shapley values ​​associated with the output of the machine learning model may contain indications of one or more features of the input data that affect the fraud detection score.

[0109] In some non-limiting embodiments or aspects, the model interpretation system 102 may determine one or more Shapley values ​​associated with the output of the machine learning model based on the inverse sign graph of the machine learning model, multiple intermediate weights, and / or multiple reference outputs (e.g., multiple reference outputs of the machine learning model stored in a cache memory location). In some non-limiting embodiments or aspects, when determining one or more Shapley values ​​associated with the output of the neural network machine learning model, the model interpretation system 102 may apply an automatic differentiation algorithm to the inverse sign graph.

[0110] In some non-limiting embodiments or aspects, the model interpretation system 102 may perform actions, such as fraud prevention procedures, transaction authorization procedures, recommendation procedures, etc., based on the output of a machine learning model and / or one or more Shapley values ​​associated with the output. For example, the model interpretation system 102 may perform actions based on determining the action to be performed after analyzing the output and / or one or more Shapley values ​​associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 may perform fraud prevention procedures associated with protecting the account of a user (e.g., a first entity, such as a user associated with user device 106) based on the output of a machine learning model and / or one or more Shapley values ​​associated with the output. For example, if the output of the machine learning model and / or one or more Shapley values ​​associated with the output (e.g., one or more Shapley values ​​associated with the output having values ​​indicating that the machine learning model correctly predicts that fraud prevention procedures are necessary) indicates that fraud prevention procedures are necessary, then the model interpretation system 102 may perform fraud prevention procedures associated with protecting the user's account. In such examples, if the output of the machine learning model and / or one or more Shapley values ​​associated with the output (e.g., one or more Shapley values ​​associated with the output that indicate the machine learning model did not correctly predict that fraud prevention procedures are necessary) indicate that fraud prevention procedures are not necessary, the model interpretation system 102 may abandon the execution of fraud prevention procedures associated with protecting the user's account.

[0111] In some non-limiting embodiments or aspects, the model interpretation system 102 may perform actions associated with the machine learning model, such as feature selection procedures, training (e.g., retraining) procedures, inference tasks (e.g., performing real-time inference tasks, such as another real-time inference task), based on the output of the machine learning model and / or one or more Shapley values ​​associated with the output. For example, the model interpretation system 102 may perform actions associated with the machine learning model based on determining the action to be performed after analyzing the output and / or one or more Shapley values ​​associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 may perform actions associated with the machine learning model based on the output of the machine learning model and / or one or more Shapley values ​​associated with the output. For example, if the output of the machine learning model and / or one or more Shapley values ​​associated with the output indicate that the action associated with the machine learning model is necessary, the model interpretation system 102 may perform fraud prevention procedures associated with protecting a user's account. In such examples, if the output of the machine learning model and / or one or more Shapley values ​​associated with the output indicate that the action associated with the machine learning model is not necessary, the model interpretation system 102 may choose not to perform the action associated with the machine learning model.

[0112] Now for reference Figures 3A-3DThis illustration shows a schematic diagram of an embodiment 300 of a process (e.g., process 200) for implementing a model-agnostic framework to provide Shapley values ​​associated with a machine learning model. In some non-limiting embodiments or aspects, one or more steps of the process may be performed (e.g., wholly, partially, etc.) by a model interpretation system 102 (e.g., one or more means of model interpretation system 102). In some non-limiting embodiments or aspects, one or more steps of the process may be performed (e.g., wholly, partially, etc.) by another means or a set of means separate from or including model interpretation system 102 (e.g., one or more means of model interpretation system 102), ML model management database 104, and / or user device 106. As shown in embodiment 300, the Shapley value can be used to interpret the difference between the output and a reference output based on the difference between the input and a corresponding reference input, which can be used to measure the importance of the target input to the output (e.g., prediction) of the machine learning model via backpropagation.

[0113] As by Figure 3A As indicated by reference numeral 305 in the accompanying drawings, the model interpretation system 102 can receive an executable file for a neural network machine learning model from the ML model management database 104. In some non-limiting embodiments or aspects, for a neural network machine learning model, t may represent the output of a neuron in an intermediate layer of the neural network machine learning model, and x0, x1, ... x n This can represent the input used to compute t from the neuron.

[0114] The reference-from-difference Δt can be expressed as Δt = t - t0, where t0 is the reference input. (For example, it may be selected based on domain knowledge and / or heuristics) the corresponding output of the neuron, and the model interpretation system 102 may assign the contribution score CΔx according to the following formula. i Δt is assigned to the constrained Δx i :

[0115]

[0116] Where CΔx i Δt is the value of x in t. i The reference difference (difference-from-reference) is the amount of reference difference.

[0117] A multiplier (e.g., a derivative) can be defined by the following formula:

[0118]

[0119] Where Δx is the reference difference in the input x, and Δt is the reference difference in the output t. In some non-limiting embodiments or aspects, since the contribution of Δx to Δt is divided by the input difference Δx, the multiplier can be used as a discrete version of the partial derivative. The chain rule for the multiplier can be defined as follows:

[0120]

[0121] Where xi is the layer H used in the neural network machine learning model. l The neuron inputs, and y0, y1, ... y n It is used for layer H l Neuron output and used for H l The input to the neurons in the subsequent layers. The analogy with partial derivatives allows calculating the contribution of the neural network machine learning model output relative to the neural network machine learning model input via backpropagation. The Shapley value can be approximated by averaging the following formula:

[0122]

[0123] Where M is the final matrix computed by the multiplier relative to the model input during backpropagation, and X is the input, and R is the reference input. This disclosure provides implementation and acceleration of the computation of M in model-agnostic frameworks for neural network machine learning models (e.g., the ONNX ecosystem). In such model-agnostic frameworks, gradient computation can be tuned for nonlinear operators (e.g., the Sigmoid operator, the MaxPooling operator, etc.) and raw gradient computation can be used for linear operators (e.g., the MatMul operator, the convolution (Conv) operator, etc.).

[0124] As by Figure 3A As further shown by reference numeral 310 in the accompanying drawings, the model interpretation system 102 can convert the format of an executable file used for a neural network machine learning model into a model-agnostic format to provide a model-agnostic file. In some non-limiting embodiments or aspects, the model interpretation system 102 can convert the format of an executable file used for a neural network machine learning model into ONNX format to provide an ONNX file for the neural network machine learning model.

[0125] As by Figure 3BAs indicated by reference numeral 315 in the accompanying drawings, the model interpretation system 102 can parse model-agnostic files used for neural network machine learning models. In some non-limiting embodiments or aspects, the model interpretation system 102 can parse model-agnostic format files (e.g., ONNX files) used for neural network machine learning models to provide forward and / or backward symbolic graphs associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the forward and / or backward symbolic graphs associated with the neural network machine learning model may contain a high-level representation of the computational flow of the neural network machine learning model. The forward and / or backward symbolic graphs may contain multiple nodes (e.g., computational nodes) and multiple edges to define the structure (e.g., architecture) and / or operations of the neural network machine learning model. In some non-limiting embodiments or aspects, each node in the forward and / or backward symbolic graphs may represent an operator (e.g., addition, multiplication, convolution, etc.), and each edge may represent a data flow between operators.

[0126] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate multiple intermediate weights and / or multiple reference outputs of a neural network machine learning model based on reference input data provided to the machine learning model. For example, the model interpretation system 102 may provide reference input data as input to the neural network machine learning model, and the neural network machine learning model may provide multiple reference outputs based on said input. Multiple intermediate weights may be generated during backpropagation when the model parameters of the neural network machine learning model are updated through forward propagation based on the reference input data.

[0127] In some non-limiting embodiments or aspects, the intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model. In the above example, the model interpretation system 102 can generate a forward symbolic graph and a backward symbolic graph associated with the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 can generate a backward symbolic graph based on the forward symbolic graph.

[0128] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a loss function for the neural network machine learning model based on the difference between the output of the forward symbolic graph and the reference input data provided to the neural network machine learning model. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a reverse symbolic graph associated with the neural network machine learning model based on the loss function used for the neural network machine learning model.

[0129] In some non-limiting embodiments or aspects, the model interpretation system 102 may compute gradients associated with the forward symbolic graph. For example, the model interpretation system 102 may compute gradients between adjacent nodes among a plurality of nodes in the forward symbolic graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges in the backward symbolic graph based on the gradients associated with the forward symbolic graph. For example, the model interpretation system 102 may generate a plurality of nodes and / or a plurality of edges in the backward symbolic graph based on the gradients between adjacent nodes among a plurality of nodes in the forward symbolic graph. In some non-limiting embodiments or aspects, the model interpretation system 102 may generate a backward symbolic graph associated with a neural network machine learning model to include a linear operator and / or a non-linear operator.

[0130] In some non-limiting embodiments or aspects, the model interpretation system 102 may first construct a forward symbolic graph. In the forward symbolic graph, a node is linked to one or more other nodes because the output of a node may be the input to another node or the output of a neural network machine learning model (e.g., a model associated with an unknown model format file, such as an ONNX model). Thus, the model interpretation system 102 may construct a backward graph, which may not yet contain a backward symbolic graph, but only a graph structure with nodes carrying information about nodes in the forward symbolic graph. The information in each node of the backward graph may include the node itself, the neighbors of the current node in the backward graph and the number of the node's neighbors, inflow and / or outflow gradients, and / or optional arguments indicating whether a gradient is passed (e.g., passgrad) to inform whether the input to the node differs from the input to the neural network machine learning model, which may be referred to as the model input.

[0131] For example, some operators such as multiplication (Mul) and addition (Add) allow two inputs to differ from the model input. When only one input differs from the model input, other operators such as matrix multiplication (MatMul) and general matrix multiplication (Gemm) allow at most two inputs to change. In some non-limiting embodiments or aspects, the model interpretation system 102 may determine whether the outflow gradient of the current node can be passed to neighboring nodes via pass grad when constructing the backpropagation graph (e.g., using a neural network resolver).

[0132] In some non-limiting embodiments or aspects, the model-agnostic framework may contain multiple operators (e.g., hundreds of operators). In some non-limiting embodiments or aspects, the model interpretation system 102 may contain gradients (e.g., for linear operators) or multipliers (e.g., for nonlinear operators) defined in the model-agnostic framework. In some non-limiting embodiments or aspects, the multiple operators may include concatenation, add, Mul, MatMul, Gemm, Sigmoid, ReLU, Softmax, Conv, MaxPool, AveragePool, GlobalAveragePool, Transpose, BatchNormalization, etc. When performing a forward sign graph, some of the resulting outputs of the gradient calculation may be stored in memory by the model interpretation system 102 for some operators. In this way, additional computation can be avoided when training a neural network machine learning model.

[0133] In some non-limiting embodiments or aspects, linear rules for linear operations can be used to compute gradients, and rescale rules and / or revealcancel rules can be used for non-linear operations to compute multipliers. In some non-limiting embodiments or aspects, operators used for gradient and / or multiplier computation using a forward sign graph include Concat, Mul, Matmul, Sigmoid, Maxpooling, GlobalMaxPooling, Avgpooling, and / or GlobalAvgPooling.

[0134] Concat can be a linear operator without local gradients. In some non-limiting embodiments or aspects, the incoming gradient can be split and / or the incoming gradient can be passed to subsequent nodes in the reverse path, depending on how the inputs of the concatenated Concat are in the forward path. The effect of Mul can be non-linear or linear, depending on whether the two inputs of the multiplication operation are differentiable with respect to the model input. If the two inputs of the multiplication operation are differentiable with respect to the model input, the model interpretation system 102 can use an explicit cancellation rule to compute the adjusted gradient. Otherwise, the model interpretation system 102 can multiply the incoming gradient with the input of the multiplication operation that is not differentiable with respect to the model input to compute the outgoing gradient. In the case of broadcasting, the smaller input can be added to the incoming gradient on the axis of the broadcast input with the larger input of the multiplication operation.

[0135] Matmul is a linear operator, and the local gradient of Matmul is the transpose of the weights relative to the input. Multiplying the local gradient by the input gradient provides the output gradient to the successor in the reverse path. Conv is a linear operator that can be used to compute the output gradient for the Conv operation. Sigmoid is a non-linear operation. The computation of the adjusted gradient is defined according to the following equation:

[0136]

[0137] Where σ(x) is the output of the Sigmoid function, x is the input of some neurons to the data we want to interpret, and r is the input of some neurons to the reference data. If xr < 1e-6, the original gradient of the Sigmoid function is returned as true, and the original gradient of the Sigmoid function can be used. Otherwise, the multiplier used for the Sigmoid function can be used by the rescaling rule. Multiplying grad* by the input gradient provides the output gradient. In some non-limiting embodiments or aspects, most activation functions obtain grad* in the same way, except for Softmax which uses an explicit cancellation rule.

[0138] Maxpooling is a non-linear operation, and the adjusted gradient used for Maxpooling can be defined by the following equation:

[0139] C = max(y x ,y r )

[0140] M x =(Cy r )×grad in

[0141] M r =(y x -C)×grad in

[0142]

[0143] Where x and r are the inputs to the max-pooling neurons for the input data and the reference input data, respectively, and y x and y r These are the outputs of these neurons. C is y x Element and y r The maximum value of the intersection between elements. Pass in the gradient grad. in Multiply by Cy r To obtain the input gradient M for cross-location x Similarly, obtain M r If xr is less than 1e-7, then the gradient is output. If the sum of the localization gradients of Maxpooling relative to x and r is zero, then the sum of the gradients is divided by xr, and this is used as the output gradient. The input gradient is propagated back to the neuron that achieves the maximum value, and all other neurons have zero gradients when the gradient used for the Maxpooling operation is computed. Note that gradients accumulate if the same neuron achieves the maximum value in different pooling windows. GlobalMaxPooling can contain a special case of Maxpooling where the pooling window size is the same as the input space. Additionally, Avgpooling is a linear operation. To compute the gradient of the input to the Avgpooling operation, the input gradient can be uniformly distributed across the pooling window, and gradients accumulate if two pooling windows overlap. Furthermore, GlobalAvgPooling is a special case of Avgpooling where the pooling window size is the same as the input space.

[0144] In some non-limiting embodiments or aspects, the automatic differentiation algorithm can be used to implement machine learning techniques, such as backpropagation (e.g., for training neural network machine learning models). In some non-limiting embodiments or aspects, the model interpretation system 102 can implement an automatic differentiation algorithm that performs a depth-first search (DFS) to identify all operators in the backward path from the model's output to its input and sums the partial gradients contributed by each operator. In some non-limiting embodiments or aspects, when using DFS, multiple types of gradient flows can be analyzed. For example, four types of gradient flows may include one2one, many2one, one2many, and many2many.

[0145] In a one2one type gradient flow, both the incoming and outgoing gradients have a branch, and the incoming gradient is multiplied by the local gradient (e.g., if any) to obtain the outgoing gradient. A one2one type gradient flow may contain an activation function, which is typical of this type of operator. If the operator has no local gradient, the incoming gradient is passed to its successor in the reverse path. In a many2one type gradient flow, there are multiple incoming gradient flows but only one outgoing gradient flow. All incoming gradients are first summed, and then the sum is multiplied by the local gradient (e.g., if any) to obtain the outgoing gradient. In a one2many type gradient flow, there is one incoming gradient flow and multiple outgoing gradient flows. After multiplying the incoming gradient by the local gradient (e.g., if any), the outgoing gradient is split or assigned to its successor. A many2many type gradient flow is a combination of many2one and one2many.

[0146] In some non-limiting embodiments, the model interpretation system 102 may use the DFS algorithm to invert the forward sign graph to compute one or more Shapley values. The following is a procedure for computing Shapley values ​​using DFS:

[0147] 1: Let S be the stack.

[0148] 2:S.push(N)

[0149] 3: Mark N as visited.

[0150] 4Define the difference-from-reference y x -y r as the loss grad in {y is the output of the model.}

[0151] 5: while S is not empty do

[0152] 6:C←S.pop()

[0153] 7:O,grad in ←F grad (C,G,grad in ){F grad is the function to compute gradients / multipliers for opera-tors.}

[0154] 8: Append O to L.

[0155] 9:for neighbor W of C in G do

[0156] 10:if W is not visited and it gets all gradient flows then

[0157] 11:S.push(W)

[0158] 12: Mark W as visited.

[0159] 13:end if

[0160] 14:end for

[0161] 15:end while

[0162] 16: return L

[0163] In the above procedure, the input includes the reverse graph G and the first computation node N, and the output includes a list of gradient nodes L. In some non-limiting embodiments or aspects, DFS takes the reverse graph G and the first computation node N as input and returns a list of computation nodes. In some non-limiting embodiments or aspects, the reverse graph G is obtained by parsing an unknown model format file used for a neural network machine learning model, and N is the first computation node in the reverse path. Each node in G contains information for performing DFS, and the name of the computation node is accessed to obtain the information. From lines 1-3, an empty stack is created, and N is pushed onto the stack, marking N as visited. Line 4 defines the loss y. x -y r This is used to compute the gradient with respect to the model input. The rest of the DFS algorithm details how to traverse all computation nodes in the reverse path. The function F grad Return the list of computed nodes O to compute the gradient of the visited node C and the grad of the next node in row 7. in The incoming gradients are processed. If a neighboring node W of C is not visited and it receives all incoming gradient streams, then W is pushed onto the stack and marked as visited. In some non-limiting embodiments or aspects, the use of an automatic differentiation algorithm (e.g., incorporating DFS) optimizes the method for generating Shapley values ​​(e.g., by caching commonly used intermediate outputs during the forward path used for backpropagation) and simplifies the computational graph for generating Shapley values ​​(e.g., a backward sign graph based on a forward sign graph).

[0164] As by Figure 3C As indicated by reference numeral 320 in the accompanying drawings, the model interpretation system 102 can receive real-time inference requests for neural network machine learning models from the user device 106. (As shown by...) Figure 3C As further shown by reference numeral 325 in the figure, the model interpretation system 102 can determine the output of the neural network machine learning model associated with the real-time inference request.

[0165] In some non-limiting embodiments or aspects, the model interpretation system 102 may use a neural network machine learning model to determine the output of the neural network machine learning model associated with the input included in the real-time inference request. For example, the model interpretation system 102 may generate scores (e.g., model scores, prediction scores, etc.) based on the inputs provided to the neural network machine learning model. In such examples, the model interpretation system 102 may generate scores based on the inputs included with the inference request and provided to the neural network machine learning model to generate scores. In some non-limiting embodiments or aspects, the score of the input (e.g., a data instance) may be equal to the average model score of the sum of the Shapley values ​​of each of the multiple features included in the input (e.g., the average model score of all inputs in a plurality of inputs).

[0166] As by Figure 3D As indicated by reference numeral 330 in the accompanying drawings, the model interpretation system 102 can determine one or more Shapley values ​​associated with the output. In some non-limiting embodiments or aspects, the model interpretation system 102 can determine one or more Shapley values ​​associated with the output of the neural network machine learning model based on the inverse sign graph of the neural network machine learning model, multiple intermediate weights, and / or multiple reference outputs (e.g., multiple reference outputs of the neural network machine learning model stored in a cache memory location). In some non-limiting embodiments or aspects, when determining one or more Shapley values ​​associated with the output of the neural network machine learning model, the model interpretation system 102 can apply an automatic differentiation algorithm to the inverse sign graph.

[0167] In some non-limiting embodiments or aspects, the model interpretation system 102 may generate (e.g., determine) a score associated with an inference task based on the output of a neural network machine learning model, the output of which is generated based on input data (e.g., input data contained in an inference request) provided as input to the neural network machine learning model. In one example, the model interpretation system 102 may generate a fraud detection score based on the output of a neural network machine learning model, and one or more Shapley values ​​associated with the output of the neural network machine learning model may contain indications of one or more features of the input data that affect the fraud detection score.

[0168] Now for reference Figure 4 The diagram illustrates a non-limiting embodiment or aspect of an exemplary environment 400 in which the methods, systems, and / or products described herein may be implemented. Figure 4 As shown, environment 400 may include a transaction service provider system 402, an issuer system 404, a client device 406, a merchant system 408, an acquirer system 410, and a communication network 412. In some non-limiting embodiments or aspects, Figure 1 Each of the model interpretation system 102, the ML model management database 104, and / or the user device 106 may be implemented by the transaction service provider system 402 (e.g., a portion thereof). In some non-limiting embodiments or aspects, Figure 1 At least one of the model interpretation system 102, ML model management database 104 and / or user device 106 may be implemented by another system, another device, another group of systems or another group of devices (e.g., a part thereof) that is separate from or includes the transaction service provider system 402, such as the issuer system 404, customer device 406, merchant system 408, acquirer system 410, etc.

[0169] Transaction service provider system 402 may include one or more devices capable of receiving and / or transmitting information to and from issuer system 404, client device 406, merchant system 408, and / or acquirer system 410 via communication network 412. For example, transaction service provider system 402 may include computing devices, such as servers (e.g., transaction processing servers), server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, transaction service provider system 402 may be associated with a transaction service provider described herein. In some non-limiting embodiments or aspects, transaction service provider system 402 may communicate with a data storage device, which may be local or remote to transaction service provider system 402. In some non-limiting embodiments or aspects, transaction service provider system 402 may be capable of receiving information from a data storage device, storing information in a data storage device, transmitting information to a data storage device, or searching for information stored in a data storage device.

[0170] The issuer system 404 may include one or more devices capable of receiving information via communication network 412 and / or transmitting information to transaction service provider system 402, client device 406, merchant system 408, and / or acquirer system 410. For example, the issuer system 404 may include computing devices, such as servers, server clusters, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 404 may be associated with the issuer institution described herein. For example, the issuer system 404 may be associated with an issuer institution that issues credit accounts, debit accounts, credit cards, debit cards, etc., to users associated with client device 406.

[0171] Client device 406 may include one or more means capable of receiving and / or transmitting information to transaction service provider system 402, issuer system 404, merchant system 408, and / or acquirer system 410 via communication network 412. Alternatively, each client device 406 may include means capable of receiving and / or transmitting information to other client devices 406 via communication network 412, another network (e.g., temporary network, local network, private network, virtual private network, etc.), and / or any other suitable communication technology. For example, client device 406 may include client-side devices, etc. In some non-limiting embodiments or aspects, client device 406 may or may not be able to receive information via short-range wireless communication connections (e.g., NFC communication connections, RFID communication connections, etc.). Communication connection Receive information via communication connections (e.g., from merchant system 408 or from another client device 406), and / or transmit information via short-range wireless communication connections (e.g., to merchant system 408).

[0172] Merchant system 408 may include one or more devices capable of receiving and / or transmitting information to and from transaction service provider system 402, issuer system 404, customer device 406, and / or acquirer system 410 via communication network 412. Merchant system 408 may also include a communication connection with customer device 406 via communication network 412 (e.g., NFC communication connection, RFID communication connection, etc.). Communication connection The merchant system 408 includes devices that receive information from the client device 406 via communication connections, etc., and / or transmit information to the client device 406 via communication network 412, communication connections, etc. In some non-limiting embodiments or aspects, the merchant system 408 may include computing devices, such as servers, server groups, client devices, client device groups, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 408 may be associated with the merchant described herein. In some non-limiting embodiments or aspects, the merchant system 408 may include one or more client devices. For example, the merchant system 408 may include client devices that allow the merchant to transmit information to the transaction service provider system 402. In some non-limiting embodiments or aspects, the merchant system 408 may include one or more devices, such as computers, computer systems, and / or peripheral devices, that the merchant can use to conduct transactions with users. For example, the merchant system 408 may include POS devices and / or POS systems.

[0173] Acquiring system 410 may include one or more devices capable of receiving and / or transmitting information to transaction service provider system 402, issuer system 404, client device 406, and / or merchant system 408 via communication network 412. For example, acquiring system 410 may include computing devices, servers, server clusters, etc. In some non-limiting embodiments or aspects, acquiring system 410 may be associated with the acquiring party described herein.

[0174] Communication network 412 may include one or more wired and / or wireless networks. For example, communication network 412 may include cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Network (PLMN), Local Area Network (LAN), Wide Area Network (WAN), Metropolitan Area Network (MAN), Telephone Network (e.g., Public Switched Telephone Network (PSTN)), Private Network (e.g., a private network associated with a transaction service provider), Temporary Network, Intranet, Internet, Fiber-based Network, Cloud Computing Network, etc., and / or combinations of these or other types of networks.

[0175] Figure 4 The number and arrangement of systems, devices, and / or networks shown are provided as examples. Figure 4 Compared to those shown, there may be additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or systems, devices, and / or networks arranged differently. Furthermore, implementation may be within a single system and / or device. Figure 4 The two or more systems or devices shown, or Figure 4 The single system or device shown may be implemented as multiple distributed systems or devices. Alternatively or additionally, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of environment 400 may perform one or more functions described as being performed by another group of systems or devices of environment 400.

[0176] Now for reference Figure 5 A diagram illustrating example components of device 500 according to a non-limiting embodiment or aspect is provided. As an example, device 500 may correspond to... Figure 1 At least one of the model interpretation system 102, ML model management database 104, and / or user device 106, and / or Figure 4At least one of the following: transaction service provider system 402, issuer system 404, client device 406, merchant system 408, and / or acquirer system 410. In some non-limiting embodiments or aspects, Figure 1 or Figure 4 Such systems or apparatuses may include at least one device 500 and / or at least one component of device 500. Figure 5 The number and arrangement of components shown are provided as examples. In some non-limiting embodiments or aspects, with Figure 5 Compared to those shown, device 500 may include additional components, fewer components, different components, or components arranged in a different manner. Alternatively, a set of components of device 500 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of device 500.

[0177] like Figure 5 As shown, device 500 may include bus 502, processor 504, memory 506, storage component 508, input component 510, output component 512, and communication interface 514. Bus 502 may include components that allow communication between components of device 500. In some non-limiting embodiments or aspects, processor 504 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 504 may include a processor (e.g., central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessor, digital signal processor (DSP), and / or any processing component that can be programmed to perform functions (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 506 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 504.

[0178] Continue to refer to Figure 5Storage component 508 may store information and / or software related to the operation and use of device 500. For example, storage component 508 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.) and / or another type of computer-readable medium. Input component 510 may include components that allow device 500 to receive information, for example, through user input (e.g., touch screen display, keyboard, keypad, mouse, buttons, switches, microphone, etc.). Alternatively or additionally, input component 510 may include sensors for sensing information (e.g., Global Positioning System (GPS) component, accelerometer, gyroscope, actuator, etc.). Output component 512 may include components that provide output information from device 500 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 514 may include transceiver-like components (e.g., transceiver, separate receiver and transmitter, etc.) that enable device 500 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 514 allows device 500 to receive information from another device and / or provide information to another device. For example, communication interface 514 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, etc. Interfaces, cellular network interfaces, etc.

[0179] Apparatus 500 can perform one or more processes described herein. Apparatus 500 can perform these processes based on processor 504 executing software instructions stored in a computer-readable medium, such as memory 506 and / or storage component 508. The computer-readable medium may contain any non-transitory memory device. The memory device includes memory space located within a single physical storage device or memory space extended across multiple physical storage devices. Software instructions may be read into memory 506 and / or storage component 508 via communication interface 514 from another computer-readable medium or from another device. When executed, the software instructions stored in memory 506 and / or storage component 508 may cause processor 504 to perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in conjunction with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software. As used herein, the term “configured to” may refer to an arrangement of software, apparatus, and / or hardware for performing and / or realizing one or more functions (e.g., actions, processes, steps of processes, etc.). For example, "a processor configured to..." can refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0180] Although embodiments have been described in detail for illustrative purposes, it should be understood that such details are for the purposes described only, and this disclosure is not limited to the disclosed embodiments or aspects, but rather is intended to cover modifications and equivalent arrangements that fall within the spirit and scope of the appended claims. For example, it should be understood that this disclosure contemplates, as far as possible, that one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method for a model interpretation system (102) deployed in an environment (400), said environment (400) comprising: The transaction service provider system (402), the issuer system (404), the merchant system (408), and the acquirer system (410) are each configured to communicate at least partially via a communication network (412). The method includes: At least one processor of the model interpretation system (102) receives an executable file for a neural network machine learning model; The executable file for the neural network machine learning model is converted into an agnostic model format using at least one processor of the model interpretation system (102) to provide an agnostic model format file for the neural network machine learning model. The model interpretation system (102) uses at least one processor to parse the agnostic model format file for the neural network machine learning model, wherein parsing the agnostic model format file for the neural network machine learning model includes: The model interpretation system (102) uses at least one processor to store multiple intermediate weights and multiple reference outputs of the neural network machine learning model in a cache memory location, wherein the multiple intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model, the reference input data including transaction data associated with multiple transactions in the environment; The model interpretation system (102) utilizes at least one processor to generate a forward symbolic graph associated with the neural network machine learning model, the forward symbolic graph defining the data flow through the neural network machine learning model during forward propagation, during which the reference input data is processed to produce a reference output; and At least one processor of the model interpretation system (102) generates a backward symbolic graph associated with the neural network machine learning model based on the forward symbolic graph, the backward symbolic graph defining how gradients are backpropagated through the neural network machine learning model to update the intermediate weights; At least one processor of the model interpretation system (102) receives a real-time inference request for the neural network machine learning model, the real-time inference request including a request to determine whether a user’s transaction in the environment is fraudulent; The model interpretation system (102) uses at least one processor to determine the output of the neural network machine learning model associated with the real-time inference request using the neural network machine learning model. Using at least one processor of the model interpretation system (102), a fraud detection score is generated based on the output; Using at least one processor of the model interpretation system (102), one or more Shapley values ​​associated with the output of the neural network machine learning model are determined based on the inverse symbol graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location. The one or more Shapley values ​​contain indications of one or more features of the input data that affect the fraud detection score. In response to the requirement that the one or more Shapley values ​​indicate that the neural network machine learning model should correctly predict fraud prevention procedures, at least one processor of the model interpretation system (102) is used to execute fraud prevention procedures associated with the protection of the user's account.

2. The computer-implemented method according to claim 1, further comprising: A loss function for the neural network machine learning model is generated based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

3. The computer-implemented method of claim 2, wherein generating the inverse symbol graph associated with the neural network machine learning model comprises: The inverse symbolic graph associated with the neural network machine learning model is generated based on the loss function used for the neural network machine learning model.

4. The computer-implemented method of claim 1, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein generating the reverse symbolic graph associated with the neural network machine learning model comprises: Calculate the gradient between adjacent nodes among the plurality of nodes in the positive symbol graph; as well as The reverse symbol graph is generated based on the gradient between adjacent nodes in the forward symbol graph.

5. The computer-implemented method of claim 1, wherein generating the inverse symbol graph associated with the neural network machine learning model comprises: Generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

6. The computer-implemented method of claim 1, wherein determining the one or more Shapley values ​​associated with the output of the neural network machine learning model comprises: The automatic differentiation algorithm is applied to the inverse symbol graph.

7. A model interpretation system (102) arranged in an environment (400), said environment (400) comprising: The transaction service provider system (402), the issuer system (404), the merchant system (408), and the acquirer system (410) are each configured to communicate at least partially via a communication network (412). The model interpretation system (102) includes: At least one processor is configured to: Receives an executable file for a neural network machine learning model; The executable file used for the neural network machine learning model is converted into an unknown model format to provide an unknown model format file for the neural network machine learning model; Parsing the agnostic model format file used for the neural network machine learning model, wherein when parsing the agnostic model format file used for the neural network machine learning model, the at least one processor is configured to: Multiple intermediate weights and multiple reference outputs of the neural network machine learning model are stored in a cache memory location, wherein the multiple intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model, the reference input data including transaction data associated with multiple transactions in the environment; Generate a forward symbol graph associated with the neural network machine learning model; the forward symbol graph defines the data flow through the neural network machine learning model during forward propagation, during which the reference input data is processed to produce a reference output; and Based on the forward symbolic graph, a backward symbolic graph associated with the neural network machine learning model is generated, wherein the backward symbolic graph defines how gradients are backpropagated through the neural network machine learning model to update the intermediate weights; Receive a real-time inference request for the neural network machine learning model, the real-time inference request including a request to determine whether a user's transaction in the environment is fraudulent; The neural network machine learning model is used to determine the output of the neural network machine learning model associated with the real-time inference request; A fraud detection score is generated based on the output; and Based on the inverse symbolic graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location, one or more Shapley values ​​associated with the output of the neural network machine learning model are determined, the one or more Shapley values ​​containing indications of one or more features of the input data that affect the fraud detection score; In response to the one or more Shapley values ​​indicating that the neural network machine learning model correctly predicts the necessity of fraud prevention procedures, fraud prevention procedures associated with the protection of the user's account are executed.

8. The model interpretation system (102) according to claim 7, wherein the at least one processor is further configured to: A loss function for the neural network machine learning model is generated based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

9. The model interpretation system (102) of claim 8, wherein when generating the inverse symbol graph associated with the neural network machine learning model, the at least one processor is configured to: The inverse symbolic graph associated with the neural network machine learning model is generated based on the loss function used for the neural network machine learning model.

10. The model interpretation system (102) of claim 7, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein when generating the reverse symbolic graph associated with the neural network machine learning model, the at least one processor is configured to: Calculate the gradient between adjacent nodes among the plurality of nodes in the forward symbol graph; and The reverse symbol graph is generated based on the gradient between adjacent nodes in the forward symbol graph.

11. The model interpretation system (102) of claim 7, wherein when generating the inverse symbol graph associated with the neural network machine learning model, the at least one processor is configured to: Generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

12. The model interpretation system (102) of claim 7, wherein when determining the one or more Shapley values ​​associated with the output of the neural network machine learning model, the at least one processor is configured to: The automatic differentiation algorithm is applied to the inverse symbol graph.

13. A computer program product for a model interpretation system (102) deployed in an environment (400), said environment (400) comprising: The transaction service provider system (402), the issuer system (404), the merchant system (408), and the acquirer system (410) are each configured to communicate at least partially via a communication network (412). The computer program product includes at least one non-transient computer-readable medium containing program instructions that, when executed by at least one processor, cause at least one processor of the model interpretation system (102) to: Receives an executable file for a neural network machine learning model; The executable file used for the neural network machine learning model is converted into an unknown model format to provide an unknown model format file for the neural network machine learning model; Parsing the agnostic model format file for the neural network machine learning model, wherein the program instructions that cause the at least one processor to parse the agnostic model format file for the neural network machine learning model cause the at least one processor to: Multiple intermediate weights and multiple reference outputs of the neural network machine learning model are stored in a cache memory location, wherein the multiple intermediate weights and multiple reference outputs of the neural network machine learning model are based on reference input data provided to the neural network machine learning model, the reference input data including transaction data associated with multiple transactions in the environment; A forward symbol graph is generated associated with the neural network machine learning model, the forward symbol graph defining the data flow through the machine learning model during forward propagation, during which the reference input data is processed to produce the reference output; as well as Based on the forward symbolic graph, a backward symbolic graph associated with the neural network machine learning model is generated, wherein the backward symbolic graph defines how gradients are backpropagated through the neural network machine learning model to update the intermediate weights; Receive a real-time inference request for the neural network machine learning model, the real-time inference request including a request to determine whether a user's transaction in the environment is fraudulent; The neural network machine learning model is used to determine the output of the neural network machine learning model associated with the real-time inference request; Using at least one processor of the model interpretation system (102), a fraud detection score is generated based on the output; Based on the inverse symbolic graph and the plurality of intermediate weights and the plurality of reference outputs of the neural network machine learning model stored in the cache memory location, one or more Shapley values ​​associated with the output of the neural network machine learning model are determined, the one or more Shapley values ​​containing indications of one or more features of the input data that affect the fraud detection score; Based on the one or more Shapley values, it is necessary for the neural network machine learning model to correctly predict fraud prevention procedures and execute fraud prevention procedures associated with the protection of the user's account.

14. The computer program product of claim 13, wherein the program instructions further cause the at least one processor to: A loss function for the neural network machine learning model is generated based on the difference between the output of the positive symbolic graph and the reference input data provided to the neural network machine learning model.

15. The computer program product of claim 14, wherein the program instructions that cause the at least one processor to generate the inverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: The inverse symbolic graph associated with the neural network machine learning model is generated based on the loss function used for the neural network machine learning model.

16. The computer program product of claim 13, wherein the forward symbolic graph comprises a plurality of nodes and a plurality of edges, and wherein the program instructions causing the at least one processor to generate the reverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: Calculate the gradient between adjacent nodes among the plurality of nodes in the forward symbol graph; and The reverse symbol graph is generated based on the gradient between adjacent nodes in the forward symbol graph.

17. The computer program product of claim 13, wherein the program instructions that cause the at least one processor to generate the inverse symbolic graph associated with the neural network machine learning model cause the at least one processor to: Generate the inverse symbol graph associated with the neural network machine learning model to include a linear operator and a nonlinear operator.

18. The computer program product of claim 13, wherein the program instructions that cause the at least one processor to determine the one or more Shapley values ​​associated with the output of the neural network machine learning model cause the at least one processor to: The automatic differentiation algorithm is applied to the inverse symbol graph.

Images