Methods, systems, and computer program products for coordinated analysis of machine learning models

Abstract

Methods, systems, and computer program products are provided for coordinated analysis of output scores and input features of machine learning models in different environments. An example method includes receiving a plurality of first data records and a plurality of second data records. A first graph is generated based on a first score generated by a machine learning model for each first data record and a second score generated by the machine learning model for each second data record. And displaying the first chart. A plurality of second charts associated with at least a subset of the plurality of features is generated. Each respective second graph is generated based on a respective first field associated with a respective feature from the first data record and a respective second field associated with a respective feature from the second data record. And displaying the second chart.

Description

Technical Field

[0001] The present disclosure generally relates to analysis of machine learning models in different environments, and in some non-limiting embodiments or aspects, to methods, systems, and computer program products for coordinated analysis of output scores and input features of machine learning models in different environments. Background Art

[0002] A machine learning model may be trained to generate certain outputs based on a data set (e.g., historical data including a plurality of historical data records). For example, a machine learning model may be configured (e.g., designed) to receive input data in a specific format and generate outputs based on the input data. When the input data is accurate and formatted correctly, the machine learning model may achieve a desired level of performance (e.g., accuracy).

[0003] However, when a machine learning model trained in one environment is deployed in another environment, the performance of the machine learning model may change (e.g., degrade, become less accurate, etc.). For example, in the training environment, the data set used to train the machine learning model may be formatted in a particular way, but once the trained machine learning model is deployed to a different environment, the input data may be inadvertently formatted at least partially differently, which may affect the performance of the machine learning model. In addition, the data set used in the training environment may be accurate, but the incoming data (e.g., real-time data) received in a different environment after the trained machine learning model is deployed may include inaccuracies, which may affect the performance of the machine learning model. Small mismatches in the original data can sometimes result in large differences in the formatted input features of the machine learning model and / or large differences in the outputs generated by the machine learning model. However, it may be difficult to determine which mismatches cause changes in the performance of the machine learning model and / or the extent to which each mismatch causes a change in the performance of the machine learning model. Summary of the invention

[0004] It is therefore an object of the present disclosure to provide a system, method, and computer program product for coordinated analysis of output scores and input features of machine learning models in different environments (e.g., which overcomes some or all of the deficiencies identified above).

[0005] According to non-limiting embodiments or aspects, a computer-implemented method for coordinating analysis of output scores and input features of machine learning models in different environments is provided. An example method may include receiving a plurality of first data records. Each first data record may include a plurality of first fields associated with a plurality of features and a first score field associated with a first score, the first score being generated by the machine learning model based on the plurality of first fields. A plurality of second data records may be received. Each second data record may include a plurality of second fields associated with the plurality of features and a second score field associated with a second score, the second score being generated by the machine learning model based on the plurality of second fields. A first chart may be generated based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records. The first chart may be displayed. A plurality of second charts associated with at least a subset of the plurality of features may be generated. Each corresponding second chart may be generated based on a corresponding first field in the plurality of first fields associated with a corresponding feature in the plurality of features and a corresponding second field in the plurality of second fields associated with the corresponding feature. The plurality of second charts may be displayed.

[0006] In some non-limiting embodiments or aspects, the plurality of features may be ranked based on at least one mismatch metric, the at least one mismatch metric being determined based on the respective first field associated with each respective feature and the respective second field associated with each respective feature. Generating the plurality of second charts may include generating the plurality of second charts associated with a selected number of the plurality of features based on the ranking of the plurality of features. Displaying the plurality of second charts may include displaying the plurality of second charts in order based on the ranking of the plurality of features.

[0007] In some non-limiting embodiments or aspects, the at least one mismatch metric may include at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

[0008] In some non-limiting embodiments or aspects, the at least one metric may include the Pearson correlation metric. Sorting the plurality of features may include: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; and / or determining the Pearson correlation metric for each respective feature based on the first difference and the respective second difference.

[0009] In some non-limiting embodiments or aspects, ranking the multiple features may include: determining a first difference based on the first score of each first data record and the second score of each second data record; determining a plurality of second differences including corresponding second differences based on the corresponding first field associated with each corresponding feature and the corresponding second field associated with each corresponding feature; and / or regressing the first difference onto the plurality of second differences based on the at least one decision tree model to provide at least one of the gain metric, the weight metric, the coverage metric, the SHAP metric, or any combination thereof.

[0010] In some non-limiting embodiments or aspects, a selection of a first point in the first graph may be received, the first point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records. A corresponding second point may be automatically selected in each second graph in the plurality of second graphs, the corresponding second point being associated with the corresponding first data record and the corresponding second data record.

[0011] In some non-limiting embodiments or aspects, a selection of a second point in one of the plurality of second graphs, the second point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records, can be received. A first point can be automatically selected in the first graph, the first point being associated with the corresponding first data record and the corresponding second data record. Additionally or alternatively, a corresponding second point can be automatically selected in each of the plurality of second graphs, but not in the one of the plurality of second graphs, the corresponding second point being associated with the corresponding first data record and the corresponding second data record.

[0012] In some non-limiting embodiments or aspects, the first chart may include a first scatter plot, and the plurality of second charts may include a plurality of second scatter plots. In some non-limiting embodiments or aspects, the first scatter plot may be converted to a first density map. Additionally or alternatively, the plurality of second scatter plots may be converted to a plurality of second density maps. The first density map may be displayed, and / or the plurality of second density maps may be displayed.

[0013] In some non-limiting embodiments or aspects, a plurality of first points in the first graph may be indexed based on a first quadtree data structure. Additionally or alternatively, a plurality of second points in each of the plurality of second graphs may be indexed based on a corresponding second quadtree data structure.

[0014] In some non-limiting embodiments or aspects, the plurality of first data records may be from a first environment. The plurality of second data records may be from a second environment different from the first environment. Each respective first data record in the plurality of first data records may have a corresponding second data record in the plurality of second data records.

[0015] In some non-limiting embodiments or aspects, the first environment may include an offline environment, and / or the second environment may include an online environment.

[0016] According to non-limiting embodiments or aspects, a system for coordinating analysis of output scores and input features of machine learning models in different environments is provided. An example system may include at least one processor and at least one non-transitory computer-readable medium including one or more instructions. The instructions, when executed by the at least one processor, may cause the at least one processor to receive a plurality of first data records. Each of the plurality of first data records may include a plurality of first fields associated with a plurality of features and a first score field, the first score field being associated with a first score generated by the machine learning model based on the plurality of first fields. The instructions, when executed by the at least one processor, may cause the at least one processor to receive a plurality of second data records. Each of the plurality of second data records may include a plurality of second fields associated with the plurality of features and a second score field, the second score field being associated with a second score generated by the machine learning model based on the plurality of second fields. The instructions, when executed by the at least one processor, may cause the at least one processor to generate a first chart based on the first score of each of the plurality of first data records and the second score of each of the plurality of second data records. The instructions, when executed by the at least one processor, may cause the at least one processor to display the first chart. The instructions, when executed by the at least one processor, may cause the at least one processor to generate a plurality of second charts associated with at least a subset of the plurality of features. Each respective second chart in the plurality of second charts may be generated based on a respective first field in the plurality of first fields associated with a respective feature in the plurality of features and a respective second field in the plurality of second fields associated with the respective feature. The instructions, when executed by the at least one processor, may cause the at least one processor to display the plurality of second charts.

[0017] In some non-limiting embodiments or aspects, the one or more instructions, when executed by the at least one processor, may cause the at least one processor to rank the plurality of features based on at least one mismatch metric, the at least one mismatch metric being determined based on the respective first field associated with each respective feature and the respective second field associated with each respective feature. Generating the plurality of second charts may include generating the plurality of second charts associated with a selected number of the plurality of features based on the ranking of the plurality of features. Displaying the plurality of second charts may include displaying the plurality of second charts in order based on the ranking of the plurality of features.

[0018] In some non-limiting embodiments or aspects, the at least one mismatch metric may include at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

[0019] In some non-limiting embodiments or aspects, the at least one metric may include the Pearson correlation metric. Sorting the plurality of features may include: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; and / or determining the Pearson correlation metric for each respective feature based on the first difference and the respective second difference.

[0020] In some non-limiting embodiments or aspects, ranking the multiple features may include: determining a first difference based on the first score of each first data record and the second score of each second data record; determining a plurality of second differences including corresponding second differences based on the corresponding first field associated with each corresponding feature and the corresponding second field associated with each corresponding feature; and / or regressing the first difference onto the plurality of second differences based on the at least one decision tree model to provide at least one of the gain metric, the weight metric, the coverage metric, the SHAP metric, or any combination thereof.

[0021] In some non-limiting embodiments or aspects, the one or more instructions, when executed by the at least one processor, may cause the at least one processor to receive a selection of a first point in the first graph, the first point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; and / or automatically select a corresponding second point in each second graph in the plurality of second graphs, the corresponding second point being associated with the corresponding first data record and the corresponding second data record.

[0022] In some non-limiting embodiments or aspects, the one or more instructions, when executed by the at least one processor, may cause the at least one processor to receive a selection of a second point in one of the plurality of second graphs, the second point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; automatically select a first point in the first graph, the first point being associated with the corresponding first data record and the corresponding second data record; and / or automatically select a corresponding second point in each of the plurality of second graphs, but not in the one of the plurality of second graphs, the corresponding second point being associated with the corresponding first data record and the corresponding second data record.

[0023] In some non-limiting embodiments or aspects, the plurality of first data records may be from a first environment, and / or the plurality of second data records may be from a second environment different from the first environment. Each respective first data record in the plurality of first data records may have a corresponding second data record in the plurality of second data records.

[0024] In some non-limiting embodiments or aspects, the first environment may include an offline environment and the second environment may include an online environment.

[0025] According to non-limiting embodiments or aspects, a computer program product is provided, the computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising one or more instructions, the one or more instructions, when executed by at least one processor, causing the at least one processor to perform any method described herein. For example, a plurality of first data records may be received. Each of the plurality of first data records may include a plurality of first fields associated with a plurality of features and a first score field, the first score field being associated with a first score generated by a machine learning model based on the plurality of first fields. A plurality of second data records may be received. Each of the plurality of second data records may include a plurality of second fields associated with the plurality of features and a second score field, the second score field being associated with a second score generated by the machine learning model based on the plurality of second fields. A first chart may be generated based on the first score of each of the plurality of first data records and the second score of each of the plurality of second data records. The first chart may be displayed. A plurality of second charts associated with at least a subset of the plurality of features may be generated. Each corresponding second chart in the plurality of second charts may be generated based on a corresponding first field in the plurality of first fields associated with a corresponding feature in the plurality of features and a corresponding second field in the plurality of second fields associated with the corresponding feature. The plurality of second charts may be displayed.

[0026] Other non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0027] Item 1: A computer-implemented method for coordinated analysis of output scores and input features of machine learning models in different environments, the method comprising: receiving a plurality of first data records using at least one processor, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field associated with a first score, the first score being generated by the machine learning model based on the plurality of first fields; receiving a plurality of second data records using at least one processor, each of the plurality of second data records comprising a plurality of second fields associated with a plurality of features and a second score field associated with a second score, the second score being generated by the machine learning model based on the plurality of second fields; generating a first chart using at least one processor based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records; displaying the first chart using at least one processor; generating a plurality of second charts associated with at least a subset of the plurality of features using at least one processor, each corresponding second chart in the plurality of second charts being generated based on a corresponding first field in the plurality of first fields associated with a corresponding feature in the plurality of features and a corresponding second field in the plurality of second fields associated with the corresponding feature; and displaying the plurality of second charts using at least one processor.

[0028] Clause 2: The method according to clause 1, further comprising: utilizing at least one processor to sort multiple features based on at least one mismatch metric, wherein the at least one mismatch metric is determined based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature, wherein generating multiple second charts comprises generating multiple second charts associated with a selected number of the multiple features based on the sorting of the multiple features, and wherein displaying the multiple second charts comprises displaying the multiple second charts in order based on the sorting of the multiple features.

[0029] Clause 3: A method according to clause 1 or clause 2, wherein at least one mismatch metric includes at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

[0030] Clause 4: A method according to any one of clauses 1 to 3, wherein at least one metric includes a Pearson correlation metric, and wherein ranking the plurality of features includes: determining a first difference based on a first score for each first data record and a second score for each second data record; determining a plurality of second differences including corresponding second differences based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature; and determining a Pearson correlation metric for each corresponding feature based on the first difference and the corresponding second difference.

[0031] Clause 5: A method according to any one of clauses 1 to 4, wherein ranking the plurality of features comprises: determining a first difference based on a first score for each first data record and a second score for each second data record; determining a plurality of second differences including corresponding second differences based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature; and regressing the first difference onto the plurality of second differences based on at least one decision tree model to provide at least one of a gain metric, a weight metric, a coverage metric, a SHAP metric, or any combination thereof.

[0032] Clause 6: A method according to any one of clauses 1 to 5, further comprising: receiving, using at least one processor, a selection of a first point in a first graph, the first point being associated with a corresponding first data record in a plurality of first data records and a corresponding second data record in a plurality of second data records; and automatically selecting, using at least one processor, a corresponding second point in each second graph in a plurality of second graphs, the corresponding second point being associated with a corresponding first data record and a corresponding second data record.

[0033] Clause 7: A method according to any one of clauses 1 to 6, further comprising: receiving, using at least one processor, a selection of a second point in one of a plurality of second charts, the second point being associated with a corresponding first data record in a plurality of first data records and a corresponding second data record in a plurality of second data records; automatically selecting, using at least one processor, a first point in a first chart, the first point being associated with a corresponding first data record and a corresponding second data record; and automatically selecting, using at least one processor, a corresponding second point in each second chart in a plurality of second charts, but not in one of the plurality of second charts, the corresponding second point being associated with a corresponding first data record and a corresponding second data record.

[0034] Clause 8: A method according to any one of clauses 1 to 7, wherein the first chart includes a first scatter plot and the plurality of second charts include a plurality of second scatter plots, the method further comprising: converting the first scatter plot into a first density plot using at least one processor; converting the plurality of second scatter plots into a plurality of second density plots using at least one processor; displaying the first density plot using at least one processor; and displaying the plurality of second density plots using at least one processor.

[0035] Clause 9: A method according to any one of clauses 1 to 8, further comprising: using at least one processor to index a plurality of first points in a first graph based on a first quadtree data structure; and using at least one processor to index a corresponding plurality of second points in each corresponding second graph in a plurality of second graphs based on a corresponding second quadtree data structure.

[0036] Clause 10: A method according to any one of clauses 1 to 9, wherein the plurality of first data records are from a first environment, wherein the plurality of second data records are from a second environment different from the first environment, and wherein each respective first data record in the plurality of first data records has a corresponding second data record in the plurality of second data records.

[0037] Clause 11: The method of any of clauses 1 to 10, wherein the first environment comprises an offline environment, and wherein the second environment comprises an online environment.

[0038] Item 12: A system for coordinated analysis of output scores and input features of machine learning models in different environments, the system comprising: at least one processor; at least one non-transitory computer-readable medium comprising one or more instructions, which when executed by the at least one processor causes the at least one processor to: receive a plurality of first data records, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field, the first score field being associated with a first score generated by the machine learning model based on the plurality of first fields; receive a plurality of second data records, each of the plurality of second data records comprising a plurality of second fields associated with a plurality of features and a second score field, the second score field being associated with a second score generated by the machine learning model based on the plurality of second fields; generate a first chart based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records; display the first chart; generate a plurality of second charts associated with at least a subset of the plurality of features, each corresponding second chart in the plurality of second charts being generated based on a corresponding first field in the plurality of first fields associated with a corresponding feature in the plurality of features and a corresponding second field in the plurality of second fields associated with the corresponding feature; and display the plurality of second charts.

[0039] Clause 13: A system according to clause 12, wherein the one or more instructions, when executed by at least one processor, cause the at least one processor to: sort the multiple features based on at least one mismatch metric, wherein the at least one mismatch metric is determined based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature, wherein generating the multiple second charts includes generating the multiple second charts associated with a selected number of the multiple features based on the sorting of the multiple features, and wherein displaying the multiple second charts includes displaying the multiple second charts in order based on the sorting of the multiple features.

[0040] Clause 14: A system according to clause 12 or clause 13, wherein at least one mismatch metric includes at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

[0041] Clause 15: A system according to any one of clauses 12 to 14, wherein at least one metric includes a Pearson correlation metric, and wherein ranking the plurality of features includes: determining a first difference based on a first score for each first data record and a second score for each second data record; determining a plurality of second differences including corresponding second differences based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature; and determining a Pearson correlation metric for each corresponding feature based on the first difference and the corresponding second difference.

[0042] Clause 16: A system according to any one of clauses 12 to 15, wherein ranking the plurality of features comprises: determining a first difference based on a first score for each first data record and a second score for each second data record; determining a plurality of second differences including respective second differences based on respective first fields associated with each respective feature and respective second fields associated with each respective feature; and regressing the first differences onto the plurality of second differences based on at least one decision tree model to provide at least one of a gain metric, a weight metric, a coverage metric, a SHAP metric, or any combination thereof.

[0043] Clause 17: A system according to any one of clauses 12 to 16, wherein one or more instructions, when executed by at least one processor, cause the at least one processor to: receive a selection of a first point in a first graph, the first point being associated with a corresponding first data record in a plurality of first data records and a corresponding second data record in a plurality of second data records; and automatically select a corresponding second point in each second graph in a plurality of second graphs, the corresponding second point being associated with a corresponding first data record and a corresponding second data record.

[0044] Clause 18: A system according to any one of clauses 12 to 17, wherein one or more instructions, when executed by at least one processor, cause the at least one processor to: receive a selection of a second point in one of a plurality of second charts, the second point being associated with a corresponding first data record in a plurality of first data records and a corresponding second data record in a plurality of second data records; automatically select a first point in the first chart, the first point being associated with a corresponding first data record and a corresponding second data record; and automatically select a corresponding second point in each of the plurality of second charts, but not in one of the plurality of second charts, the corresponding second point being associated with a corresponding first data record and a corresponding second data record.

[0045] Clause 19: A system according to any one of clauses 12 to 18, wherein the plurality of first data records are from a first environment, wherein the plurality of second data records are from a second environment different from the first environment, wherein each respective first data record in the plurality of first data records has a corresponding second data record in the plurality of second data records, and wherein the first environment comprises an offline environment and the second environment comprises an online environment.

[0046] Clause 20: A computer program product, the computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising one or more instructions, the one or more instructions, when executed by at least one processor, causes the at least one processor to: receive a plurality of first data records, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field, the first score field being associated with a first score generated by a machine learning model based on the plurality of first fields; receive a plurality of second data records, each of the plurality of second data records comprising a plurality of second fields associated with a plurality of features and a second score field, the second score field being associated with a second score generated by the machine learning model based on the plurality of second fields; generate a first chart based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records; display the first chart; generate a plurality of second charts associated with at least a subset of the plurality of features, each respective second chart in the plurality of second charts being generated based on a respective first field in the plurality of first fields associated with a respective feature in the plurality of features and a respective second field in the plurality of second fields associated with the respective feature; and display the plurality of second charts.

[0047] Clause 21: A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising one or more instructions which, when executed by at least one processor, cause the at least one processor to perform a method according to any one of clauses 1 to 11.

[0048] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of the related structural elements and combinations of parts and economies of manufacture, will become more apparent when considering the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Additional advantages and details are explained in more detail below with reference to non-limiting exemplary embodiments shown in the schematic drawings, in which:

[0050] Figure 1 is a schematic diagram of an example system for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0051] Figure 2 is a flow chart of an example process for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0052] Figure 3 are example graphs of output scores for machine learning models in different environments according to some non-limiting embodiments or aspects;

[0053] Figure 4A and 4B An example diagram showing an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0054] Figures 5A to 5C An example graphical user interface showing an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0055] Figure 6 An example quadtree according to an example implementation of a process for coordinated analysis of output scores and input features of machine learning models in different environments is presented according to some non-limiting embodiments or aspects;

[0056] Fig. 7A and 7B An example scatter plot and an example density plot are respectively presented according to an example implementation of a process for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0057] Figures 8A to 8C An example scatter plot showing an example implementation of a process for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects;

[0058] Fig. 9 is a diagram of an example environment in which the methods, systems, and / or computer program products described herein may be implemented according to some non-limiting embodiments or aspects;

[0059] Fig.10 is a schematic diagram of example components of an apparatus for use in conjunction with some non-limiting embodiments or aspects. DETAILED DESCRIPTION

[0060] For the purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the orientation of the embodiments in the accompanying drawings. However, it shall be understood that the embodiments may employ various alternative variations and step sequences, except where expressly specified to the contrary. It shall also be understood that the specific devices and processes illustrated in the accompanying drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0061] The aspects, parts, elements, structures, actions, steps, functions, instructions, etc. used herein should not be understood as critical or necessary unless explicitly described as such. Moreover, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, a combination of related items and unrelated items, etc.), and can be used interchangeably with "one or more" or "at least one". In the case of wishing only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" etc. is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part on".

[0062] As used herein, the term "acquirer" may refer to an entity that is licensed and / or approved by a transaction service provider to initiate transactions (e.g., payment transactions) using a payment device associated with the transaction service provider. Transactions that may be initiated by an acquirer may include payment transactions (e.g., purchases, original letter of credit transactions (OCTs), account funding transactions (AFTs), etc.). In some non-limiting embodiments or aspects, the acquirer may be a financial institution, such as a bank. As used herein, the term "acquirer system" may refer to one or more computing devices operated by or on behalf of an acquirer, such as a server computer executing one or more software applications.

[0063] As used herein, the term "account identifier" may include one or more PANs, tokens, or other identifiers associated with a customer account. The term "token" may refer to an identifier used as a replacement or substitute identifier for an original account identifier such as a PAN. An account identifier may be alphanumeric or any combination of characters and / or symbols. A token may be associated with a PAN or other original account identifier in one or more data structures (e.g., one or more databases, etc.) so that the token can be used to conduct transactions without directly using the original account identifier. In some examples, an original account identifier such as a PAN may be associated with multiple tokens for different individuals or purposes.

[0064] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transfer, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, etc.) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, etc.) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit.

[0065] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, etc. The computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. The computing device may also be a desktop computer or other form of non-mobile computer.

[0066] As used herein, the terms "electronic wallet" and "electronic wallet application" refer to one or more electronic devices and / or software applications configured to initiate and / or conduct payment transactions. For example, an electronic wallet may include a mobile device executing an electronic wallet application, and may also include server-side software and / or databases for maintaining and providing transaction data to the mobile device. An "electronic wallet provider" may include an entity that provides and / or maintains electronic wallets for customers, such as Google Android Apple Samsung And / or other similar electronic payment systems. In some non-limiting examples, the issuing bank may be an electronic wallet provider.

[0067] As used herein, the term "issuer institution" may refer to one or more entities, such as a bank, that provide accounts to customers for conducting transactions (e.g., payment transactions), such as initiating credit and / or debit payments. For example, an issuer institution may provide customers with account identifiers, such as a primary account number (PAN), that uniquely identify one or more accounts associated with the customer. The account identifier may be implemented on a portable financial device, such as a physical financial instrument (e.g., a payment card), and / or may be electronic and used for electronic payments. The term "issuer system" refers to one or more computer devices operated by or on behalf of an issuer institution, such as a server computer executing one or more software applications. For example, an issuer system may include one or more authorization servers for authorizing transactions.

[0068] As used herein, the term "merchant" may refer to a person or entity that provides goods and / or services, or access to goods and / or services, to a customer based on a transaction, such as a payment transaction. The term "merchant" or "merchant system" may also refer to one or more computer systems operated by or on behalf of a merchant, such as a server computer that executes one or more software applications.

[0069] As used herein, a "point of sale (POS) device" may refer to one or more devices that can be used by a merchant to conduct a transaction (e.g., a payment transaction) and / or process a transaction. For example, a POS device may include one or more client devices. Additionally or alternatively, a POS device may include a peripheral device, a card reader, a scanning device (e.g., a code scanner), Communication receiver, near field communication (NFC) receiver, radio frequency identification (RFID) receiver and / or other contactless transceiver or receiver, contact-based receiver, payment terminal, etc. As used herein, a "point of sale (POS) system" may refer to one or more client devices and / or peripheral devices used by merchants to conduct transactions. For example, a POS system may include one or more POS devices, and / or other similar devices that can be used to conduct payment transactions. In some non-limiting embodiments or aspects, a POS system (e.g., a merchant POS system) may include one or more server computers that are programmed or configured to process online payment transactions through web pages, mobile applications, etc.

[0070] As used herein, the terms "client" and / or "client device" may refer to one or more client-side devices or systems (e.g., remote from a transaction service provider system) used to initiate or facilitate a transaction (e.g., a payment transaction). As an example, a "client device" may refer to one or more POS devices used by a merchant, one or more acquirer host computers used by an acquirer, one or more mobile devices used by a user, and the like. In some non-limiting embodiments or aspects, a client device may be an electronic device configured to communicate with one or more networks and initiate or facilitate a transaction. For example, a client device may include one or more computers, portable computers, laptop computers, tablet computers, mobile devices, cellular phones, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), PDAs, and the like. In addition, a "client" may also refer to an entity (e.g., a merchant, an acquirer, etc.) that owns, utilizes, and / or operates a client device for initiating a transaction (e.g., for initiating a transaction with a transaction service provider).

[0071] As used herein, the term "payment device" may refer to a payment card (e.g., a credit or debit card), a gift card, a smart card, smart media, a payroll card, a healthcare card, a wristband, a machine-readable medium containing account information, a key chain device or fob, an RFID transponder, a retailer discount or membership card, a cellular telephone, an electronic wallet mobile application, a personal digital assistant (PDA), a pager, a security card, a computing device, an access card, a wireless terminal, a transponder, etc. In some non-limiting embodiments or aspects, a payment device may include volatile or non-volatile memory to store information (e.g., an account identifier, an account holder's name, etc.).

[0072] As used herein, the term "payment gateway" may refer to an entity and / or a payment processing system operated by or on behalf of such an entity, which provides payment services (e.g., transaction service provider payment services, payment processing services, etc.) to one or more merchants (e.g., transaction service provider payment services, payment processing services, etc.). The payment service may be associated with the use of a portable financial device managed by a transaction service provider. As used herein, the term "payment gateway system" may refer to one or more computer systems, computer devices, servers, server groups, etc. operated by or on behalf of a payment gateway.

[0073] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that there may be various other arrangements. In addition, multiple computing devices (e.g., servers, point of sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system." As used herein, references to "servers" or "processors" may refer to previously described servers and / or processors, different servers and / or processors, and / or combinations of servers and / or processors stated to perform previous steps or functions. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different servers and / or processors stated to perform a second step or function.

[0074] As used herein, the term "transaction service provider" may refer to an entity that receives transaction authorization requests from merchants or other entities and, in some cases, provides payment assurance through an agreement between the transaction service provider and the issuer organization. For example, a transaction service provider may include, for example The term "transaction processing system" may refer to one or more computer systems operated by or on behalf of a transaction service provider, such as a transaction processing server executing one or more software applications. A transaction processing server may include one or more processors and, in some non-limiting embodiments or aspects, may be operated by or on behalf of a transaction service provider.

[0075] Non-limiting embodiments or aspects of the disclosed subject matter relate to methods, systems, and computer program products for analyzing machine learning models in different environments, including but not limited to coordinating the analysis of output scores and input features of machine learning models in different environments. For example, non-limiting embodiments or aspects of the disclosed subject matter provide receiving a plurality of first data records (e.g., a first data set from a first environment, such as an offline environment), each first data record including a plurality of first fields associated with a plurality of features and a first score field associated with a first score generated by the machine learning model based on the first field, and receiving a plurality of second data records (e.g., a second data set from a second environment, such as an online environment), each second data record second field including a plurality of second fields associated with (the same) plurality of features and a second score field associated with a second score generated by the (same) machine learning model based on the second field. For example, each respective first data record may have a corresponding second data record (e.g., representing the same thing, but having data from a different environment). A first chart can be generated based on the first score of each first data record and the second score of each second data record, and a plurality of second charts associated with at least a subset of the plurality of features can be generated (e.g., each respective second chart can be generated based on the respective first field associated with the respective feature and the respective second field associated with the (same) respective feature in the plurality of second fields). The first chart and the plurality of second charts can be displayed (e.g., simultaneously). Thus, the disclosed subject matter allows for coordinated, synchronized analysis between the output scores of machine learning models in different environments and the input features of the different environments. In addition, non-limiting embodiments or aspects of the disclosed subject matter rank the plurality of features based on at least one mismatch metric, the at least one mismatch metric being determined based on the respective first field associated with each respective feature and the respective second field associated with each respective feature, and a second chart can be generated and / or displayed based on the ranking (e.g., in order of the largest to smallest mismatch metric). With the aid of this ranking and display in the ranked order, relevant data records for which there are a large number of mismatches (and which may result in differences in the behavior and / or performance of the machine learning model) can be effectively identified and visualized. In addition, non-limiting embodiments or aspects of the disclosed subject matter provide for receiving a selection of one or more points in one of the displayed graphs, and automatically selecting (e.g., highlighting, changing color and / or shading, etc.) corresponding points (e.g., associated with the same data record) in each of the other graphs. With this dynamic coordinated selection, mismatched input features that contribute to different machine learning model behaviors and / or performances can be discovered. In addition, non-limiting embodiments or aspects of the disclosed subject matter provide indexing points of the graphs based on a quadtree data structure, which can more efficiently automatically select corresponding points in different graphs when selecting points in one of the graphs.

[0076] Reference now Figure 1 , depicts an example system 100 for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. For example, the system 100 may include a model analysis system 102, at least one database (e.g., a first database 104-1 and a second database 104-2, collectively referred to as "database 104" and individually referred to as "database 104"), a user device 106, and / or a communication network 112.

[0077] The model analysis system 102 may include one or more devices capable of receiving information from and / or transmitting information to a database 104 and / or a user device 106 (e.g., directly via a wired or wireless communication connection, indirectly via a communication network 112, etc.). For example, the model analysis system 102 may include a computing device, such as a server, a server cluster, a desktop computer, a portable computer, a mobile device, and / or other similar devices. In some non-limiting embodiments or aspects, the model analysis system 102 may communicate with a data storage device (e.g., one or more databases 104), which may be local or remote to the model analysis system 102. In some non-limiting embodiments or aspects, the model analysis system 102 may be capable of receiving information from, storing information in, transmitting information to, or searching for information stored in a data storage device (e.g., one or more databases 104).

[0078] Each database 104 may include one or more devices capable of receiving information from the model analysis system 102 and / or the user device 106 and / or transmitting information to the model analysis system and / or the user device (e.g., directly via a wired or wireless communication connection, indirectly via a communication network 112, etc.). For example, each database 104 may include a computing device, such as a server, a server group, a desktop computer, a portable computer, a mobile device, and / or other similar devices. Additionally or alternatively, each user database 104 may include a device capable of receiving information from other user databases 104 and / or transmitting information to the other user databases (e.g., directly via a wired or wireless communication connection, indirectly via a communication network 112, etc.). In some non-limiting embodiments or aspects, each database 104 may include a data storage device. In some non-limiting embodiments or aspects, each database 104 may be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device. In some non-limiting embodiments or aspects, at least one database 104 may be part of the model analysis system 102 and / or part of the same system as the model analysis system 102 .

[0079] In some non-limiting embodiments or aspects, the system 100 may include at least two databases 104 storing data records from at least two environments. For example, the first database 104-1 may store a plurality of first data records from a first environment (e.g., an offline environment, such as a development, training, and / or testing environment), as described herein. Additionally or alternatively, the second database 104-2 may store a plurality of second data records from a second environment different from the first environment (e.g., an online environment, such as an inference time environment, a runtime environment, an electronic payment processing network environment, etc.), as described herein. In some non-limiting embodiments or aspects, each respective first data record in the plurality of first data records may have a corresponding second data record in the plurality of second data records, as described herein.

[0080] The user device 106 may include one or more devices capable of receiving information from and / or transmitting information to the model analysis system 102 and / or database 104 (e.g., directly via a wired or wireless communication connection, indirectly via a communication network 112, etc.). For example, the user device 106 may include a computing device, such as a mobile device, a portable computer, a desktop computer, and / or other similar devices. Additionally or alternatively, each user device 106 may include a device capable of receiving information from and / or transmitting information to other user devices 106 (e.g., directly via a wired or wireless communication connection, indirectly via a communication network 112, etc.). In some non-limiting embodiments or aspects, the user device 106 may be part of the model analysis system 102 and / or part of the same system as the model analysis system 102. For example, the model analysis system 102, the database 104, and the user device 106 may all be a single system and / or a single computing device (and / or a portion thereof).

[0081] The communication network 112 may include one or more wired and / or wireless networks. For example, the communication network 112 may include a cellular network (e.g., Long Term Evolution). networks, third generation (5G) networks, fourth generation (4G) networks, fifth generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks (e.g., private networks associated with transaction service providers), ad hoc networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0082] Figure 1 The number and arrangement of systems and / or devices shown in the FIG. are provided as examples. Figure 1There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner than the systems and / or devices shown in the drawings. Furthermore, a single system and / or device may be implemented Figure 1 Two or more systems or devices shown in Figure 1 The single system or device shown in the embodiment may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0083] Reference now Figure 2 , showing a process 200 for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. Figure 2 The steps shown in are for example purposes only. It should be understood that in non-limiting embodiments or aspects, additional, fewer, different and / or different order of steps may be used. In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed by the model analysis system 102 (e.g., one or more devices of the model analysis system 102) (e.g., completely, partially, etc.). In some non-limiting embodiments or aspects, one or more steps of process 200 may be performed by another system, another device, another system group, or another device group that is independent of or includes the model analysis system 102, such as the database 104 and / or the user device 106 (e.g., completely, partially, etc.).

[0084] like Figure 2 As shown, at step 202, process 200 may include receiving a first data record. For example, model analysis system 102 may receive a plurality of first data records from database 104 (eg, from first database 104-1).

[0085] In some non-limiting embodiments or aspects, each first data record may include multiple first fields associated with multiple features and a first score field associated with a first score, where the first score is generated by the machine learning model based on the multiple first fields.

[0086] In some non-limiting embodiments or aspects, the first data record may be from a first environment. For example, the first environment may include an offline environment. For purposes of illustration, the offline environment may include at least one of a development environment, a training environment, a test environment, any combination thereof, and the like.

[0087] In some non-limiting embodiments or aspects, the first database 104-1 can be associated with a first environment. In some non-limiting embodiments or aspects, the first database 104-1 can include a table format of multiple rows and multiple columns to store first data records. Each row can be associated with a corresponding first data record (e.g., can represent the corresponding first data record). Each column can be associated with a corresponding feature. Each row can include multiple first fields, and the multiple first fields include a corresponding first field for each corresponding feature (e.g., a corresponding field for each column). For illustrative purposes, Table 1 includes example data from a first (e.g., offline) environment:

[0088] Table 1

[0089] ID Feature 1 (Classification) Feature 2 (Numerical value) Feature 3 (Embedding) Score 1 1 0.02 0.1235 0.2 2 1 0.4 0.3453 0.9 3 6 0.124 0.2356 0.5 … … … … …

[0090] As shown in Table 1, each first data record may include an identifier (ID) in an identifier field. In addition, each data record may include multiple feature fields. For example, each feature may be of a corresponding type (e.g., categorical, numerical, embedded, etc.). In some non-limiting embodiments or aspects, there may be multiple feature fields of at least one type (e.g., multiple feature fields of each feature type). Each data record may also include a score field, which may include a score generated by a machine learning model based on multiple feature fields of the data record.

[0091] In some non-limiting embodiments or aspects, the machine learning model can be trained and / or tested in the first environment. For example, the machine learning model can be trained by generating an output (e.g., a score) for each first data record (or a subset thereof) (e.g., by the model analysis system 102 and / or at least one other computing device and / or system, which can be independent of or include the model analysis system 102). For illustrative purposes, the value of the feature field of each first data record (or a subset thereof) can be input to the machine learning model (e.g., by the model analysis system 102 and / or at least one other computing device / system) to generate a corresponding output (e.g., a score) for each corresponding first data record (e.g., by forward propagation, etc.). The loss of each output can be determined (e.g., based on comparison and / or difference with a label and / or expected output / score, based on a loss function, based on an error calculation, etc.). In some non-limiting embodiments or aspects, parameters (e.g., weights, connection values, etc.) of the machine learning model can be updated (e.g., adjusted) based on the loss (e.g., by back-propagation, gradient calculation, etc.) (e.g., by the model analysis system 102 and / or at least one other computing device / system).

[0092] like Figure 2As shown, at step 204, process 200 may include receiving a second data record. For example, model analysis system 102 may receive a plurality of second data records from database 104 (eg, from second database 104-2).

[0093] In some non-limiting embodiments or aspects, each of the plurality of second data records may include a plurality of second fields associated with a plurality of features (e.g., a plurality of features nominally the same as the first data record) and a second score field associated with a second score, the second score being generated by a machine learning model (e.g., the same machine learning model as used for the first data record) based on the plurality of second fields.

[0094] In some non-limiting embodiments or aspects, the plurality of second data records may be from a second environment different from the first environment. For example, the second environment may include an online environment. For purposes of illustration, the online environment may include at least one of an inference time environment, a runtime environment, an electronic payment processing network environment, any combination thereof, and the like.

[0095] In some non-limiting embodiments or aspects, each corresponding first data record may have a corresponding second data record in a plurality of second data records. For example, the corresponding second data record may have an identifier (ID) that matches the ID of the corresponding first data record. In some non-limiting embodiments or aspects, the corresponding second data record may represent the same thing as the corresponding first data record, but the data of the second data record (e.g., the value of the feature field) may be from a different environment.

[0096] In some non-limiting embodiments or aspects, the second data record may be based on data received in a second environment (e.g., in real time). For example, a message (e.g., a data packet, a transaction message, an authorization request, an authorization response, a settlement message, a clearing message, etc.) including data (e.g., raw data) associated with the second data record may be received (e.g., by the model analysis system 102 and / or at least one other computing device and / or system, which may be independent of or include the model analysis system 102). In some non-limiting embodiments or aspects, the value of each feature field of each second data record (e.g., the value associated with each feature) may be determined (e.g., derived, calculated, aggregated, etc.) based on the (raw) data in the message (e.g., by feature engineering, etc.). In some non-limiting embodiments or aspects, the value of each feature field of each second data record may be expected to be the same as the corresponding feature field in the corresponding first data record. However, there may be unintentional differences between the first data record and the corresponding second data record. For example, numerical feature fields may be calculated with different levels of precision (e.g., different numbers of significant digits, rounding to different decimal places, etc.). Additionally or alternatively, the classification feature field may be different and / or incorrect in the second environment compared to the first environment. Additionally or alternatively, the embedded feature field (e.g., which may be determined based on other feature fields and / or raw data) may have different values ​​based on changes in other feature fields and / or raw data.

[0097] For purposes of illustration, Table 2 includes example data from a second (eg, online) environment:

[0098] Table 2

[0099] ID Feature 1 (Classification) Feature 2 (Numerical value) Feature 3 (Embedding) Score 1 1 0.02 0.1235 0.2 2 1 0.41 0.3453 0.5 3 60 0.124 0.6234 0.1 … … … … …

[0100] As shown in Table 2, each second data record may include an identifier (ID) in an identifier field. In addition, each data record may include multiple feature fields. For example, each feature may belong to a corresponding type (e.g., classification, numerical, embedding, etc.). In some non-limiting embodiments or aspects, there may be multiple feature fields of at least one type (e.g., multiple feature fields of each feature type). Each data record may also include a score field, which may include a score generated by a machine learning model based on multiple feature fields of the data record.

[0101] like Figure 2 As shown, at step 206, process 200 may include generating a first graph. For example, model analysis system 102 may generate the first graph based on a first score for each of the plurality of first data records and a second score for each of the plurality of second data records.

[0102] In some non-limiting embodiments or aspects, the first graph may include a plurality of points (e.g., first points). For example, each respective point may be associated with a respective first data record and a corresponding second data record (e.g., the second data record having an identifier that matches the identifier of the respective first data record).

[0103] In some non-limiting embodiments or aspects, each corresponding point may have at least one coordinate based on a first score from a corresponding first data record and at least one coordinate based on a second score from a corresponding second data record. For example, a first chart may include a vertical axis associated with a first score (e.g., generated by a machine learning model based on a first data record from a first environment and / or a first field thereof) and a horizontal axis associated with a second score (e.g., generated by a machine learning model based on a second data record from a second environment and / or a second field thereof), as described herein. Thus, each corresponding point of the chart may have a vertical coordinate based on a first score from a corresponding first data record and a horizontal coordinate based on a second score from a corresponding second data record, as described herein. In some non-limiting embodiments or aspects, the vertical axis and the horizontal axis may be switched so that the vertical axis is associated with the second score and the horizontal axis is associated with the first score (and thus, the vertical coordinate may be based on the second score and the horizontal coordinate may be based on the first score).

[0104] In some non-limiting embodiments or aspects, the first graph can be a scatter plot, as described herein.

[0105] In some non-limiting embodiments or aspects, the first graph can be a density graph, as described herein.

[0106] like Figure 2 As shown in FIG. 2 , at step 208 , process 200 can include displaying a first chart. For example, model analysis system 102 and / or user device 106 can display the first chart.

[0107] For the purpose of illustration, reference is now made to Figure 3 And continue to refer to Figure 2 , showing an example chart 300 of output scores of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the chart 300 can be displayed by the user device 106 (e.g., included in a graphical user interface (GUI) displayed thereon) and / or the model analysis system 102 (e.g., a computing device thereof), as described herein. In some non-limiting embodiments or aspects, the chart 300 can be displayed by another system, another device, another group of systems, or another group of devices independent of or including the user device 106 and / or the model analysis system 102, as described herein.

[0108] like Figure 3 As shown, the chart 300 may include a vertical axis associated with a first score (e.g., generated by a machine learning model based on a first data record and / or a first field from a first environment) and a horizontal axis associated with a second score (e.g., generated by a machine learning model based on a second data record and / or a second field from a second environment). Each point 322 of the chart may have a vertical coordinate based on a first score from a corresponding first data record and a horizontal coordinate based on a second score from a corresponding second data record, as described herein. In some non-limiting embodiments or aspects, the vertical axis and the horizontal axis may be switched such that the vertical axis is associated with the second score and the horizontal axis is associated with the first score (and therefore, the vertical coordinate may be based on the second score and the horizontal coordinate may be based on the first score).

[0109] In some non-limiting embodiments or aspects, when the first score of the first data record and the second score of the corresponding second data record are substantially equal, the points 322 plotted based on those scores will be substantially along the diagonal line of the chart 300 (e.g., in the matching area 324). For example, the point 322-1 may correspond to the first data record in Table 1 having an ID value of 1 and the second data record in Table 2 having an ID value of 1. Thus, the point 322-1 may have a vertical coordinate of 0.2 and a horizontal coordinate of 0.2, and be located along the diagonal line of the chart 300 (e.g., in the matching area 324).

[0110] In some non-limiting embodiments or aspects, when the first score of the first data record and the second score of the corresponding second data record are substantially unequal (e.g., do not match), the points 322 plotted based on those scores will not be substantially along the diagonal of the chart 300. Instead, such points 322 will be located in one of the mismatch regions 326 (e.g., above or below the diagonal). For example, point 322-2 may correspond to a first data record in Table 1 having an ID value of 2 and a second data record in Table 2 having an ID value of 2. Thus, point 322-2 may have a vertical coordinate of 0.9 and a horizontal coordinate of 0.5 and be located above the diagonal of the chart 300 (e.g., in the upper mismatch region 326). For example, point 322-3 may correspond to a first data record in Table 1 having an ID value of 3 and a second data record in Table 2 having an ID value of 3. Thus, point 322 - 3 may have a vertical coordinate of 0.5 and a horizontal coordinate of 0.1 and be located above the diagonal line of graph 300 (eg, in upper mismatch region 326 ).

[0111] Reference again Figure 2At step 210, process 200 may include generating a second graph. For example, model analysis system 102 may generate a plurality of second graphs associated with at least a subset of the plurality of features. For example, each respective second graph may be generated based on a respective first field in the plurality of first fields associated with a respective feature in the plurality of features and a respective second field in the plurality of second fields associated with the (same) respective feature.

[0112] In some non-limiting embodiments or aspects, each second graph may include a plurality of points (e.g., second points). For example, each respective point may be associated with a respective first data record and a corresponding second data record (e.g., the second data record having an identifier that matches an identifier of the respective first data record).

[0113] In some non-limiting embodiments or aspects, each corresponding point may have at least one coordinate based on a first field associated with a corresponding feature from a corresponding first data record and at least one coordinate based on a second field associated with a corresponding feature from a corresponding second data record. For example, each second chart may include a vertical axis associated with a first field (e.g., a first data record from a first environment) and a horizontal axis associated with a second field (e.g., a second data record from a second environment), as described herein. Thus, each corresponding point of the chart may have a vertical coordinate based on a first field from a corresponding first data record and a horizontal coordinate based on a second field from a corresponding second data record, as described herein. In some non-limiting embodiments or aspects, the vertical axis and the horizontal axis may be switched so that the vertical axis is associated with the second field and the horizontal axis is associated with the first field (and therefore, the vertical coordinate may be based on the second field, and the horizontal coordinate may be based on the first field).

[0114] In some non-limiting embodiments or aspects, when a first field of a first data record and a second field of a corresponding second data record are substantially equal, the points plotted based on those scores will be substantially along a diagonal line of the graph (e.g., in a matching region, as described herein). When a first field of a first data record and a second field of a corresponding second data record are substantially unequal (e.g., do not match), the points plotted based on those scores will not be substantially along a diagonal line of the graph. Instead, such points will be located above or below the diagonal line (e.g., in one of the non-matching regions, as described herein).

[0115] In some non-limiting embodiments or aspects, each second graph can be a scatter plot, as described herein.

[0116] In some non-limiting embodiments or aspects, each second graph can be a density graph, as described herein.

[0117] like Figure 2 As shown, at step 212, process 200 can include displaying a second chart. For example, model analysis system 102 can display a plurality of second charts.

[0118] In some non-limiting embodiments or aspects, the second graph can be displayed simultaneously with the first graph. Additionally or alternatively, like the first graph, the second graph can be part of (e.g., at least partially displayed in) the same GUI (and / or accessible in the same window).

[0119] For the purpose of illustration, reference is now made to Figure 4A and 4B And continue to refer to Figure 2 , showing example graphs (e.g., score graph 401 and feature graphs 402-1 to 402-4) according to an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, Figure 4A and 4B The charts shown in can be displayed by the user device 106 (e.g., included in a GUI displayed thereon) and / or the model analysis system 102 (e.g., its computing device). In some non-limiting embodiments or aspects, Figure 4A and 4B The charts shown in may be displayed by another system, another device, another group of systems, or another group of devices independent of or including the user device 106 and / or the model analysis system 102 .

[0120] In some non-limiting embodiments or aspects, score graph 401 can be the same as or similar to the first graph, as described herein (e.g., with reference to Figure 2 In some non-limiting embodiments or aspects, each feature graph 402-1 to 402-4 may be the same as or similar to the second graph, as described herein (e.g., with reference to Figure 2 Steps 210 and 212).

[0121] like Figure 4A As shown in , score graph 401 may include a vertical axis, a horizontal axis, and points 422 based on a first score for each first data record and a second score for each second data record, as described herein.

[0122] Feature graph 402-1 can be associated with a first feature (e.g., feature 1), as described herein. Feature graph 402-1 can include a vertical axis, a horizontal axis, and a point 432-1 based on a first field from each first data record associated with the first feature and a second field from a corresponding second data record associated with the first feature, as described herein.

[0123] Feature graph 402-2 can be associated with a second feature (e.g., feature 2), as described herein. Feature graph 402-2 can include a vertical axis, a horizontal axis, and a point 432-2 based on a first field from each first data record associated with the second feature and a second field from a corresponding second data record associated with the second feature, as described herein.

[0124] Feature graph 402-3 can be associated with a third feature (e.g., feature 3), as described herein. Feature graph 402-3 can include a vertical axis, a horizontal axis, and a point 432-3 based on a first field from each first data record associated with the third feature and a second field from a corresponding second data record associated with the third feature, as described herein.

[0125] Feature graph 402-4 can be associated with a fourth feature (e.g., feature 4), as described herein. Feature graph 402-4 can include a vertical axis, a horizontal axis, and a point 432-4 based on a first field from each first data record associated with the fourth feature and a second field from a corresponding second data record associated with the fourth feature, as described herein.

[0126] Continue to refer Figure 2 In some non-limiting embodiments or aspects, a selection of one or more points in one of the displayed graphs may be received (e.g., by the user device 106 and / or the model analysis system 102). Corresponding points (e.g., associated with the same data record) may be automatically selected (e.g., by highlighting, changing color, and / or shading, etc.) in each of the other graphs (e.g., by the user device 106 and / or the model analysis system 102).

[0127] For illustrative purposes, Figure 4B As shown, a selection (e.g., input indicating a selection, such as a selection box 403, a lasso, a click, a tap, etc.) of at least one point 423 in the score graph 401 may be received (e.g., by a user device 106 and / or a model analysis system 102). For example, each selected point 423 may be associated with a corresponding first data record (e.g., from a first environment) and a corresponding second data record (e.g., from a second environment). At least one corresponding point 433-1 to 433-4 may be automatically selected in each feature graph 402-1 to 402-4 (e.g., by a user device 106 and / or a model analysis system 102). For example, as with the corresponding selected point 423, each automatically selected corresponding point 433-1 to 433-4 may be associated with the same first data record and second data record.

[0128] In some non-limiting embodiments or aspects, a selection (e.g., input indicating a selection, such as a selection box, a lasso, a click, a tap, etc.) of at least one point in one of the feature graphs 402-1 to 402-4 may be received (e.g., by the user device 106 and / or the model analysis system 102). For example, each selected point may be associated with a corresponding first data record (e.g., from a first environment) and a corresponding second data record (e.g., from a second environment). At least one corresponding point may be automatically selected in the score graph 401 (e.g., by the user device 106 and / or the model analysis system 102), and / or at least one corresponding point may be automatically selected in each of the other feature graphs 402-1 to 402-4 (e.g., as the corresponding point, the corresponding point may be associated with the same first data record and second data record).

[0129] Continue to refer Figure 2 In some non-limiting embodiments or aspects, the plurality of features may be ranked (e.g., by the model analysis system 102 and / or the user device 106) based on at least one mismatch metric determined based on a respective first field associated with each respective feature in a first data record (e.g., from a first environment) and a respective second field associated with each respective feature in a second data record (e.g., from a second environment). For example, generating a second chart (e.g., a feature chart) may include generating a second chart associated with a selected number of the plurality of features based on the ranking of the plurality of features (e.g., generating a second chart with a selected number having a maximum mismatch metric, etc.). Additionally or alternatively, displaying the second chart may include displaying the second chart in order based on the ranking of the plurality of features (e.g., in order from the maximum mismatch metric to the minimum mismatch metric).

[0130] In some non-limiting embodiments or aspects, at least one mismatch metric may include at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

[0131] In some non-limiting embodiments or aspects, at least one mismatch metric may include a Pearson correlation metric. Sorting the plurality of features may include determining (e.g., by the model analysis system 102 and / or the user device 106) a first difference based on a first score for each first data record and a second score for each second data record, and / or determining (e.g., by the model analysis system 102 and / or the user device 106) a plurality of second differences including corresponding second differences based on corresponding first fields associated with each corresponding feature and corresponding second fields associated with each corresponding feature. A Pearson correlation metric for each corresponding feature may be determined (e.g., by the model analysis system 102 and / or the user device 106) based on the first difference and the corresponding second difference. For purposes of illustration, the mismatch metric (Corr ) may be determined based on the following equation. i ):

[0132] Corr i =Pearson(|D1(score)-D2(score)|,|D1(f i )-D2(f i )|),

[0133] Where D1 is the set of the first data record, D2 is the set of the second data record, score is the score field, and f i is the corresponding field associated with the corresponding feature, and Pearson is a function used to calculate the Pearson correlation coefficient based on the two differences in the equation.

[0134] In some non-limiting embodiments or aspects, at least one metric may include determining (e.g., by the model analysis system 102 and / or the user device 106) a first difference based on a first score for each first data record and a second score for each second data record, and / or determining (e.g., by the model analysis system 102 and / or the user device 106) a plurality of second differences including corresponding second differences based on a corresponding first field associated with each corresponding feature and a corresponding second field associated with each corresponding feature. The first difference may be regressed onto the plurality of second differences (e.g., by the model analysis system 102 and / or the user device 106) based on at least one decision tree model to provide at least one of a gain metric, a weight metric, a coverage metric, a SHAP metric, or any combination thereof. For example, at least one decision tree model may include a plurality of decision tree models (e.g., an ensemble decision tree model). For purposes of illustration, at least one (e.g., ensemble) decision tree model may include an XGBoost model.

[0135] For the purpose of illustration, reference is now made to Figures 5A to 5C And continue to refer to Figure 2, showing an example GUI 500 according to an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the GUI 500 can be displayed by the user device 106 and / or the model analysis system 102 (e.g., a computing device thereof). In some non-limiting embodiments or aspects, the GUI 500 can be displayed by another system, another device, another group of systems, or another group of devices that is independent of or includes the user device 106 and / or the model analysis system 102.

[0136] like Figure 5A As shown in , GUI 500 may include at least one of a first area 541, a second area 542, and a third area 543. For example, the first area 541 may include (e.g., having displayed therein) a score graph 501 (e.g., a first chart, as described herein). The second area 542 may include (e.g., having displayed therein) a plurality of feature graphs 502 (e.g., a plurality of second charts, as described herein). The third area 543 may include at least one graphical element for configuring GUI 500 and / or its charts (e.g., score graphs 501 and / or feature graphs 502) (e.g., configuring the generation, display, and / or sorting of charts). For example, the graphical element may include a check box 551, a text box 552, a check box 553, a drop-down menu 554, and / or a button 555.

[0137] In some non-limiting embodiments or aspects, selecting one or more check boxes 551 may cause the GUI 500 to display a feature map 502 of a selected number of features of the corresponding (one or more) types (e.g., categorical features, numerical features, and / or embedded features). Unchecking (or leaving unchecked) any check box 551 may cause the GUI 500 to not display features of the corresponding type. The selected number of features of each type (and therefore the number of corresponding feature maps 502 displayed) may be determined based on the number entered into the corresponding text box 552. For example, Figure 5A As shown, the check box 551 associated with the numerical feature is selected, and the selected number in the corresponding text box 552 is 80, so the GUI 500 can include a maximum of 80 feature graphs 502 associated with the numerical feature. (Although Figure 5A 542 , but other feature graphs 502 may be visible if GUI 500 is displayed on a larger screen or if a user scrolls down past feature graphs 502 displayed in second region 542 of GUI 500. )

[0138] In some non-limiting embodiments or aspects, selecting checkbox 553 may cause GUI 500 to display score map 501 and / or feature map 502 as a scatter plot. Unchecking (or leaving unchecked) checkbox 553 may cause GUI 500 to display score map 501 and / or feature map 502 as a different type of chart (e.g., density map, etc.).

[0139] In some non-limiting embodiments or aspects, selecting button 555 may cause GUI 500 (and / or a computing device displaying GUI 500) to export data associated with any selected point (e.g., data records associated with any selected point) in a graph (e.g., score graph 501 and / or feature graph 502). For example, the exported data may be formatted as tabular data, such as a spreadsheet (e.g., comma separated values ​​(CSV file), Excel file, etc.), a table, etc.

[0140] In some non-limiting embodiments or aspects, selecting different options in the drop-down menu 554 may cause the GUI 500 (and / or a computing device displaying the GUI 500) to rank the features based on different mismatch metrics and / or to display the feature graph 502 according to the ranking. Figure 5B As shown, a correlation metric (e.g., a Pearson correlation metric) may be selected from a drop-down menu 554. Figure 5C As shown, the second chart 502 is re-ordered and displayed according to the ranked order of the features, the ranking being based on the selected relevance metric.

[0141] Continue to refer Figure 2 , in some non-limiting embodiments or aspects, the chart may include a scatter plot. For example, a first chart (e.g., a score chart) may include a first scatter plot, and multiple second charts (e.g., feature charts) may include multiple second scatter plots. In some non-limiting embodiments or aspects, the scatter plot may be converted to another type of chart (e.g., a density chart). For example, the model analysis system 102 and / or the user device 106 may convert the first scatter plot into a first density chart and / or convert the multiple second scatter plots into multiple second density charts. Additionally or alternatively, the model analysis system 102 and / or the user device 106 may display the first density chart and / or the second density chart.

[0142] For the purpose of illustration, reference is now made to Fig. 7A and 7B And continue to refer to Figure 2, respectively showing an example scatter plot 700a and an example density plot 700b according to an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the scatter plot 700a and the density plot 700b can be displayed by a user device 106 (e.g., included in a GUI displayed thereon) and / or a model analysis system 102 (e.g., a computing device thereof). In some non-limiting embodiments or aspects, the scatter plot 700a and the density plot 700b can be displayed by another system, another device, another group of systems, or another group of devices that is independent of or includes the user device 106 and / or the model analysis system 102.

[0143] In some non-limiting embodiments or aspects, the model analysis system 102 and / or the user device 106 may convert the scatter plot 700a into a density plot 700b (e.g., in response to input from a user, such as selection or deselection of a graphical element of a GUI as described herein).

[0144] Continue to refer Figure 2 , the points in each graph can be indexed (e.g., to enable more efficient automatic selection of corresponding points in different graphs when selecting a point in one of the graphs). For example, the points in each graph can be indexed based on a quadtree data structure. For example, the model analysis system 102 and / or the user device 106 can index a plurality of first points in a first graph (e.g., a score graph) based on a first quadtree data structure. Additionally or alternatively, the model analysis system 102 and / or the user device 106 can index a plurality of corresponding second points in each corresponding second graph (e.g., a feature graph) based on a corresponding second quadtree data structure.

[0145] For the purpose of illustration, reference is now made to Figure 6 And continue to refer to Figure 2 , showing an example quadtree 600 according to an example implementation of a process for coordinated analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, the quadtree 600 can be implemented by the model analysis system 102 (e.g., a computing device thereof) and / or the user device 106. In some non-limiting embodiments or aspects, the quadtree 600 can be implemented by another system, another device, another group of systems, or another group of devices that is independent of or includes the model analysis system 102 and / or the user device 106.

[0146] like Figure 2As shown, points 622 (e.g., of a scatter plot) may be plotted. For example, the model analysis system 102 and / or the user device 106 may determine a quadtree data structure by partitioning the plotted points 622 based on the partition points 662 to produce leaf nodes 664 (e.g., areas in the graph partitioned by the partition points 662), each leaf node having a number of points less than a target number of points 622 (e.g., less than five points 622). For example, the entire graph may be referred to as the root of the quadtree data structure. The model analysis system 102 and / or the user device 106 may partition the graph (e.g., the root) into four sections (e.g., quadrilaterals), and each quadrilateral may be referred to as a branch of the quadtree data structure. The model analysis system 102 and / or the user device 106 may repeatedly partition each branch until the number of points 622 in each partition is less than the target number of points 622. Each partition having less than the target number of points 622 may be a leaf node 664. In some non-limiting embodiments or aspects, each point 622 may be indexed based on the leaf node 664 containing the point 622 and / or based on the partition point 662 used to generate the leaf node 664 .

[0147] In some non-limiting embodiments or aspects, each graph (eg, scatter plot) may have its own quadtree data structure.

[0148] In some non-limiting embodiments or aspects, automatically selecting one of points 622 (e.g., in response to selecting a corresponding point in another graph, as described herein) may include searching a quadtree data structure of the graph to determine (e.g., identify) point 622 in the graph.

[0149] Continue to refer Figure 2 , data associated with any selected point in a graph (e.g., score graph 501 and / or feature graph 502) (e.g., first data record and / or second data record associated with any selected point) can be exported (e.g., by model analysis system 102 and / or user device 106). For example, the exported data can be formatted as tabular data, such as a spreadsheet (e.g., comma separated values ​​(CSV file), Excel file, etc.), a table, etc.

[0150] In some non-limiting embodiments or aspects, data (e.g., associated with any selected point) may be exported (e.g., by the model analysis system 102 and / or the user device 106) in response to receiving input from a user (e.g., via a graphical element of a GUI), as described herein. In some non-limiting embodiments or aspects, data (e.g., associated with any selected point) may be automatically exported (e.g., by the model analysis system 102 and / or the user device 106) based on a mismatch metric satisfying (e.g., exceeding) a threshold. For example, the model analysis system 102 and / or the user device 106 may determine that a mismatch metric associated with a feature satisfies (e.g., exceeds) a threshold, and the model analysis system 102 and / or the user device 106 may automatically export data associated with the feature (e.g., first and / or second data records and / or fields thereof) in response to determining that the mismatch metric satisfies the threshold.

[0151] In some non-limiting embodiments or aspects, exporting data may include transmitting (e.g., by the model analysis system 102 and / or the user device 106) at least one message (e.g., a notification message) to at least one user (e.g., a creator, developer, trainer, and / or user responsible for deploying the machine learning model).

[0152] In some non-limiting embodiments or aspects, at least one action (e.g., a remedial action) can be taken (e.g., by the model analysis system 102 and / or the user device 106) based on a mismatch between the first score and the second score, based on a mismatch associated with at least one feature, based on a mismatch metric associated with at least one feature (e.g., a mismatch metric that satisfies a threshold), etc. For example, the action can include conducting an investigation, reformatting at least one field of the first data record and / or the second data record, exporting data (e.g., as described herein), and / or transmitting at least one message, as described herein. For example, conducting an investigation can include determining a root cause of the mismatch. For example, the root cause can be determined based on at least one pattern of points of a graph, and the pattern can depend on the type of feature represented by the graph.

[0153] For the purpose of illustration, reference is now made to Figures 8A to 8C And continue to refer to Figure 2 , showing an example scatter plot according to an example implementation of a process for coordinating analysis of output scores and input features of machine learning models in different environments according to some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, Figures 8A to 8C The chart shown in can be displayed by the user device 106 (e.g., included in a GUI displayed thereon) and / or the model analysis system 102 (e.g., its computing device). In some non-limiting embodiments or aspects, Figures 8A to 8CThe graphs shown in may be displayed by another system, another device, another group of systems, or another group of devices independent of or including the user device 106 and / or the model analysis system 102 .

[0154] like Fig. 8A As shown, scatter plot 800a can be associated with a classification feature. The pattern observed in scatter plot 800a may include selected points 823 that are not aligned with other points 822 (e.g., the feature has at least one misaligned point). Therefore, the root cause of the mismatch in scatter plot 800a can include additional categories included in one of the data sets (e.g., the first data record or the second data record), which are not included in the other of the data sets.

[0155] like Figure 8B As shown, scatter plot 800b can be associated with a numerical feature (e.g., a transaction speed metric). The pattern observed in scatter plot 800b may include a selected point 823 having a larger mismatch metric than other points 822 (e.g., the feature has a larger mismatch at small values). Therefore, the root cause of the mismatch in scatter plot 800b may include this particular numerical feature (e.g., transaction speed metric) being very sensitive when the value is small.

[0156] like Figure 8C As shown, scatter plot 800c can be associated with an embedding feature (e.g., the embedding can include a vector representation of the original data and can have a smaller size than the original data). The pattern observed in scatter plot 800c can include selected points 823 and / or other points 822 around the diagonal that appear to have a random amount of mismatch (e.g., random values ​​around 0). Therefore, the root cause of the mismatch in scatter plot 800c can include that a separate model (e.g., an upstream machine learning model responsible for determining the embedding based on the original data) was not properly initialized and / or refreshed (e.g., the cache was not refreshed).

[0157] Reference now Fig. 9 , depicts a diagram of a non-limiting embodiment or aspect of an exemplary environment 900 in which the systems, products, and / or methods described herein may be implemented. Fig. 9 As shown, environment 900 may include a transaction service provider system 902, an issuer system 904, a client device 906, a merchant system 908, an acquirer system 910, and a communication network 912. In some non-limiting embodiments or aspects, Figure 1 Each of the model analysis system 102, database 104, and / or user device 106 may be implemented by the transaction service provider system 902 (e.g., a portion thereof). In some non-limiting embodiments or aspects, Figure 1At least one of the model analysis system 102, database 104 and / or user device 106 may be implemented by another system, another device, another system group or another device group (e.g., a portion thereof) that is independent of or includes the transaction service provider system 902, such as the issuer system 904, the customer device 906, the merchant system 908, the acquirer system 910, etc.

[0158] The transaction service provider system 902 may include one or more devices capable of receiving information from an issuer system 904, a client device 906, a merchant system 908, and / or an acquirer system 910 via a communication network 912 and / or transmitting information to the issuer system, the client device, the merchant system, and / or the acquirer system. For example, the transaction service provider system 902 may include a computing device, such as a server (e.g., a transaction processing server, etc.), a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the transaction service provider system 902 may be associated with a transaction service provider described herein. In some non-limiting embodiments or aspects, the transaction service provider system 902 may communicate with a data storage device, which may be local or remote to the transaction service provider system 902. In some non-limiting embodiments or aspects, the transaction service provider system 902 may be capable of receiving information from the data storage device, storing information in the data storage device, transmitting information to the data storage device, or searching for information stored in the data storage device.

[0159] The issuer system 904 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 902, the client device 906, the merchant system 908, and / or the acquirer system 910 via the communication network 912. For example, the issuer system 904 may include a computing device, such as a server, a server group, and / or other similar devices. In some non-limiting embodiments or aspects, the issuer system 904 may be associated with an issuer organization described herein. For example, the issuer system 904 may be associated with an issuer organization that issues a credit account, a debit account, a credit card, a debit card, etc. to a user associated with the client device 906.

[0160] The client devices 906 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 902, the issuer system 904, the merchant system 908, and / or the acquirer system 910 via the communication network 912. Additionally or alternatively, each client device 906 may include a device capable of receiving information from and / or transmitting information to other client devices 906 via the communication network 912, another network (e.g., an ad hoc network, a local network, a private network, a virtual private network, etc.), and / or any other suitable communication technology. For example, the client devices 906 may include client devices, etc. In some non-limiting embodiments or aspects, the client devices 906 may or may not be capable of communicating via a short-range wireless communication connection (e.g., an NFC communication connection, an RFID communication connection, Communication connection, communication connection, etc.) (e.g., from merchant system 908 or from another client device 906), and / or transmit information via a short-range wireless communication connection (e.g., to merchant system 908).

[0161] The merchant system 908 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 902, the issuer system 904, the client device 906, and / or the acquirer system 910 via the communication network 912. The merchant system 908 may also include a communication connection (e.g., an NFC communication connection, an RFID communication connection, a NFC communication connection, a ...RFID communication connection, a NFC communication connection, a RFID communication connection, a RFID communication connection, a NFC communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, a RFID communication connection, Communication connection, In some non-limiting embodiments or aspects, the merchant system 908 may include a computing device, such as a server, a server group, a client device, a client device group, and / or other similar devices. In some non-limiting embodiments or aspects, the merchant system 908 may be associated with the merchant described herein. In some non-limiting embodiments or aspects, the merchant system 908 may include one or more client devices. For example, the merchant system 908 may include a client device that allows the merchant to transmit information to the transaction service provider system 902. In some non-limiting embodiments or aspects, the merchant system 908 may include one or more devices that can be used by merchants to conduct transactions with users, such as computers, computer systems, and / or peripheral devices. For example, the merchant system 908 may include a POS device and / or a POS system.

[0162] The acquirer system 910 may include one or more devices capable of receiving information from and / or transmitting information to the transaction service provider system 902, the issuer system 904, the client device 906, and / or the merchant system 908 via the communication network 912. For example, the acquirer system 910 may include a computing device, a server, a server group, etc. In some non-limiting embodiments or aspects, the acquirer system 910 may be associated with an acquirer as described herein.

[0163] The communication network 912 may include one or more wired and / or wireless networks. For example, the communication network 912 may include a cellular network (e.g., Long Term Evolution). networks, third generation (5G) networks, fourth generation (4G) networks, fifth generation (5G) networks, code division multiple access (CDMA) networks, etc.), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks (e.g., private networks associated with transaction service providers), ad hoc networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0164] supply Fig. 9 The number and arrangement of systems, devices, and / or networks shown are examples. Additional systems, devices, and / or networks; fewer systems, devices, and / or networks; different systems, devices, and / or networks; and / or Fig. 9 The systems, devices and / or networks shown are arranged differently. In addition, they can be implemented within a single system or device. Fig. 9 Two or more systems or devices shown, or Fig. 9 The single system or device shown may be implemented as multiple distributed systems or devices. Additionally or alternatively, a system set (e.g., one or more systems) and / or a device set (e.g., one or more devices) of environment 900 may perform one or more functions described as being performed by another system set or another device set of environment 900.

[0165] Reference now Fig.10 , showing a diagram of example components of a device 1000 according to a non-limiting embodiment or aspect. The device 1000 may correspond to Figure 1 Model analysis system 102, database 104 and / or user device 106 and / or Fig. 9Transaction service provider system 902, issuer system 904, client device 906, merchant system 908, acquirer system 910, and / or communication network 912. In some non-limiting embodiments or aspects, such systems or devices may include at least one device 1000 and / or at least one component of device 1000. The number and arrangement of components shown are provided as examples. In some non-limiting embodiments or aspects, device 1000 may include at least one device 1000 and / or at least one component of device 1000. Fig.10 Components shown in the figure may include additional components, fewer components, different components, or differently arranged components than those shown in the figure. Additionally or alternatively, a set of components (e.g., one or more components) of device 1000 may perform one or more functions described as being performed by another set of components of device 1000.

[0166] like Fig.10 As shown, the device 1000 may include a bus 1002, a processor 1004, a memory 1006, a storage component 1008, an input component 1010, an output component 1012, and a communication interface 1014. The bus 1002 may include components that allow communication between components of the device 1000. In some non-limiting embodiments or aspects, the processor 1004 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 1004 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.). The memory 1006 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 1004.

[0167] Continue to refer Fig.10, the storage component 1008 may store information and / or software related to the operation and use of the device 1000. For example, the storage component 1008 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 1010 may include a component that allows the device 1000 to receive information, for example, via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 1010 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 1012 may include a component that provides output information from the device 1000 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 1014 may include a transceiver-like component (e.g., a transceiver, a separate receiver and a transmitter, etc.) that enables the device 1000 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 1014 may allow the device 1000 to receive information from another device and / or provide information to another device. For example, the communication interface 1014 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, Interface, cellular network interface, etc.

[0168] Device 1000 can perform one or more processes described herein. Device 1000 can perform these processes based on processor 1004 executing software instructions stored by computer-readable media such as memory 1006 and / or storage component 1008. Computer-readable media may include any non-transient memory device. The memory device includes a memory space located inside a single physical storage device or a memory space spread over multiple physical storage devices. Software instructions can be read from another computer-readable medium or from another device to memory 1006 and / or storage component 1008 via communication interface 1014. When executed, the software instructions stored in memory 1006 and / or storage component 1008 can cause processor 1004 to perform one or more processes described herein. In addition or alternatively, hard-wired circuits can be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuits and software. The term "programmed or configured" used herein refers to the arrangement of software, hardware circuit systems, or any combination thereof on one or more devices.

[0169] Although the embodiments have been described in detail for the purpose of illustration, it should be understood that such details are used for that purpose only, and the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.

Claims

1. A computer-implemented method for coordinating analysis of output scores and input features of machine learning models in different environments, the method comprising: Receiving, with at least one processor, a plurality of first data records, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field associated with a first score, the first score being generated by a machine learning model based on the plurality of first fields; receiving, with at least one processor, a plurality of second data records, each of the plurality of second data records comprising a plurality of second fields associated with the plurality of features and a second score field associated with a second score generated by the machine learning model based on the plurality of second fields; generating, with at least one processor, a first graph based on the first score for each first data record in the plurality of first data records and the second score for each second data record in the plurality of second data records; displaying the first graph using at least one processor; generating, with at least one processor, a plurality of second graphs associated with at least a subset of the plurality of features, each respective second graph in the plurality of second graphs being generated based on a respective first field in the plurality of first fields associated with a respective feature in the plurality of features and a respective second field in the plurality of second fields associated with the respective feature; and The second plurality of graphs are displayed using at least one processor.

2. The method according to claim 1, further comprising: rank the plurality of features based on at least one mismatch metric determined based on the respective first field associated with each respective feature and the respective second field associated with each respective feature, using at least one processor, wherein generating the plurality of second graphs comprises generating the plurality of second graphs associated with a selected number of the plurality of features based on the ranking of the plurality of features, and Wherein displaying the plurality of second charts comprises displaying the plurality of second charts in order based on the ranking of the plurality of features.

3. The method of claim 2, wherein the at least one mismatch metric comprises at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on at least one decision tree model, a coverage metric determined based on at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on at least one decision tree model, or any combination thereof.

4. The method of claim 3, wherein the at least one metric comprises the Pearson correlation metric, and wherein ranking the plurality of features comprises: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; and The Pearson correlation measure for each respective feature is determined based on the first difference and the respective second difference.

5. The method of claim 3, wherein ranking the plurality of features comprises: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; as well as The first difference is regressed onto the plurality of second differences based on the at least one decision tree model to provide at least one of the gain metric, the weight metric, the coverage metric, the SHAP metric, or any combination thereof.

6. The method according to claim 1, further comprising: receiving, with at least one processor, a selection of a first point in the first graph, the first point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; as well as A respective second point is automatically selected, using at least one processor, in each second graph of the plurality of second graphs, the respective second point being associated with the respective first data record and the respective second data record.

7. The method according to claim 1, further comprising: receiving, with at least one processor, a selection of a second point in one of the plurality of second graphs, the second point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; automatically selecting, using at least one processor, a first point in the first graph, the first point being associated with the corresponding first data record and the corresponding second data record; as well as A respective second point is automatically selected, using at least one processor, in each of the plurality of second graphs other than the one of the plurality of second graphs, the respective second point being associated with the respective first data record and the respective second data record.

8. The method of claim 1, wherein the first graph comprises a first scatter graph and the plurality of second graphs comprises a plurality of second scatter graphs, the method further comprising: converting the first scatter plot into a first density plot using at least one processor; converting the plurality of second scatter plots into a plurality of second density plots using at least one processor; displaying the first density map using at least one processor; as well as The plurality of second density maps are displayed using at least one processor.

9. The method according to claim 1, further comprising: indexing, using at least one processor, a plurality of first points in the first graph based on a first quadtree data structure; as well as A respective plurality of second points in each respective second graph in the plurality of second graphs is indexed based on a respective second quadtree data structure, utilizing at least one processor.

10. The method of claim 1, wherein the plurality of first data records are from a first environment, wherein the plurality of second data records are from a second environment different from the first environment, and Wherein each respective first data record of the plurality of first data records has a corresponding second data record of the plurality of second data records.

11. The method of claim 10, wherein the first environment comprises an offline environment, and wherein the second environment comprises an online environment.

12. A system for coordinating analysis of output scores and input features of machine learning models in different environments, the system comprising: at least one processor; as well as at least one non-transitory computer-readable medium comprising one or more instructions that, when executed by the at least one processor, cause the at least one processor to: Receiving a plurality of first data records, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field associated with a first score generated by a machine learning model based on the plurality of first fields; receiving a plurality of second data records, each of the plurality of second data records comprising a plurality of second fields associated with the plurality of features and a second score field associated with a second score generated by the machine learning model based on the plurality of second fields; generating a first graph based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records; displaying the first chart; generating a plurality of second graphs associated with at least a subset of the plurality of features, each respective second graph of the plurality of second graphs being generated based on a respective first field of the plurality of first fields associated with a respective feature of the plurality of features and a respective second field of the plurality of second fields associated with the respective feature; and The plurality of second graphs are displayed.

13. The system of claim 12, wherein the one or more instructions, when executed by the at least one processor, cause the at least one processor to: ranking the plurality of features based on at least one mismatch metric determined based on the respective first field associated with each respective feature and the respective second field associated with each respective feature, wherein generating the plurality of second graphs comprises generating the plurality of second graphs associated with a selected number of the plurality of features based on the ranking of the plurality of features, and Wherein displaying the plurality of second charts comprises displaying the plurality of second charts in order based on the ranking of the plurality of features.

14. The system of claim 13, wherein the at least one mismatch metric comprises at least one of a correlation metric, a Pearson correlation metric, a gain metric determined based on at least one decision tree model, a weight metric determined based on the at least one decision tree model, a coverage metric determined based on the at least one decision tree model, a Shapley Additive Explanations (SHAP) metric determined based on the at least one decision tree model, or any combination thereof.

15. The system of claim 14, wherein the at least one metric comprises the Pearson correlation metric, and wherein ranking the plurality of features comprises: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; and The Pearson correlation measure for each respective feature is determined based on the first difference and the respective second difference.

16. The system of claim 14, wherein ranking the plurality of features comprises: determining a first difference based on the first score for each first data record and the second score for each second data record; determining a plurality of second differences including respective second differences based on the respective first field associated with each respective feature and the respective second field associated with each respective feature; as well as The first difference is regressed onto the plurality of second differences based on the at least one decision tree model to provide at least one of the gain metric, the weight metric, the coverage metric, the SHAP metric, or any combination thereof.

17. The system of claim 12, wherein the one or more instructions, when executed by the at least one processor, cause the at least one processor to: receiving a selection of a first point in the first graph, the first point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; and A respective second point is automatically selected in each second graph of the plurality of second graphs, the respective second point being associated with the respective first data record and the respective second data record.

18. The system of claim 12, wherein the one or more instructions, when executed by the at least one processor, cause the at least one processor to: receiving a selection of a second point in one of the plurality of second graphs, the second point being associated with a corresponding first data record in the plurality of first data records and a corresponding second data record in the plurality of second data records; automatically selecting a first point in the first graph, the first point being associated with the corresponding first data record and the corresponding second data record; as well as A respective second point is automatically selected in each of the plurality of second graphs but not in the one of the plurality of second graphs, the respective second point being associated with the respective first data record and the respective second data record.

19. The system of claim 1, wherein the plurality of first data records are from a first environment, wherein the plurality of second data records are from a second environment different from the first environment, wherein each respective first data record in the plurality of first data records has a corresponding second data record in the plurality of second data records, and The first environment includes an offline environment, and the second environment includes an online environment.

20. A computer program product comprising at least one non-transitory computer readable medium comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: Receiving a plurality of first data records, each of the plurality of first data records comprising a plurality of first fields associated with a plurality of features and a first score field associated with a first score generated by a machine learning model based on the plurality of first fields; receiving a plurality of second data records, each of the plurality of second data records comprising a plurality of second fields associated with the plurality of features and a second score field associated with a second score generated by the machine learning model based on the plurality of second fields; generating a first graph based on the first score of each first data record in the plurality of first data records and the second score of each second data record in the plurality of second data records; displaying the first chart; generating a plurality of second graphs associated with at least a subset of the plurality of features, each respective second graph of the plurality of second graphs being generated based on a respective first field of the plurality of first fields associated with a respective feature of the plurality of features and a respective second field of the plurality of second fields associated with the respective feature; and The plurality of second graphs are displayed.

Images