System and methods for fairness evaluation in modelled systems

Abstract

In one example, a method includes building a first model including a first data set, the first data set excluding protected class. A second model is built including a second data set, the second data set including a subset of the first data set. The models may be used to generate and index score distributions. The distributions are compared to determine a self-report correlation. In response to determining the aggregate correlation is less than the first disparate impact threshold, the method generates a first prediction of disparate impact. The method then generates an aggregate correlation for attributes in the first model with the self-reported protected class attribute, the plurality of attributes being generated by the first model. In response to determining the aggregate correlation exceeds the second disparate impact threshold, the method generates a second prediction of disparate impact.

Description

SYSTEM AND METHODS FOR FAIRNESS EVALUATION IN MODELLED SYSTEMSCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 587,895 filed October 4, 2023, and entitled “SYSTEM AND METHOD FOR FAIRNESS EVALUATION IN MODELLED SYSTEMS,” the entire contents of which is incorporated herein by reference in its entirety for all purposes.Technical Field

[0002] The present disclosure relates generally to artificial intelligence. More specifically, but not by way of limitation, this disclosure relates to systems and methods for fairness evaluation in modelled systems.Background

[0001] Artificial Intelligence (Al) and Machine Learning (ML) systems used in various industries have requirements for compliance with explainability to demonstrate fairness in the AI / ML systems. Currently, there is a lack of fairness and bias testing available for these industries due to flaws in data available for such testing. That is, the data used to demonstrate fairness, or a lack of bias may unintentionally introduce false positive results of disparate impact.Summary

[0003] Various aspects of the present disclosure provide systems and methods for evaluating fairness and bias in modelled systems. In one example, a method includes building a first model including a first data set, where the first data set excludes protected class data as a predictor value. A second model is built including a second data set, where the second data set includes a subset of the first data set and where the second data set is further defined by one self-reported protected class attribute. The models may be used to generate and index a first score distribution from the first model and a second score distribution from the second model. The first and second score distributions are comparedto determine a self-report correlation. The self-report correlation is compared against a first disparate impact threshold to determine whether the self-report correlation is less than the first disparate impact threshold, and if so, generates a first prediction of disparate impact. The method then generates an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, where the plurality of attributes being generated by the first model. The aggregate correlation is compared against a second disparate impact threshold. In response to determining the aggregate correlation exceeds the second disparate impact threshold, the method generates a second prediction of disparate impact.

[0004] In another example, a system includes a processing device and a memory device in which instructions executable by the processing device are stored for causing the processing device to determine predictions of disparate impact. The system includes first model including a first data set, where the first data set excludes protected class data as a predictor value. A second model includes a second data set, where the second data set includes a subset of the first data set and where the second data set is further defined by one self-reported protected class attribute. The models may be used to generate and index a first score distribution from the first model and a second score distribution from the second model. The system compares the score distributions to determine a self-report correlation. The self-report correlation is compared against a first disparate impact threshold to determine whether the self-report correlation is less than the first disparate impact threshold, and if so, the system generates a first prediction of disparate impact. The system then generates an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, where the plurality of attributes being generated by the first model. The system compares the aggregate correlation against a second disparate impact threshold. In response to determining the aggregate correlation exceeds the second disparate impact threshold, the system generates a second prediction of disparate impact.

[0005] In yet another example, a non-transitory computer-readable medium having program code that is executable by a processor device to cause computing device toperform operations is described. The operations include building a first model including a first data set, where the first data set excludes protected class data as a predictor value. A second model is built including a second data set, where the second data set includes a subset of the first data set and where the second data set is further defined by one selfreported protected class attribute. The models may be used to generate and index a first score distribution from the first model and a second score distribution from the second model. The first and second score distributions are compared to determine a self-report correlation. The self-report correlation is compared against a first disparate impact threshold to determine whether the self-report correlation is less than the first disparate impact threshold, and if so, generates a first prediction of disparate impact. Further operations then generate an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, where the plurality of attributes being generated by the first model. The aggregate correlation is compared against a second disparate impact threshold. In response to determining the aggregate correlation exceeds the second disparate impact threshold, a second prediction of disparate impact is generated.

[0006] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim.

[0007] The foregoing, together with other features and examples, will become more apparent upon referring to the following specification, claims, and accompanying drawings.Brief Description of the Drawings

[0008] FIG. 1 is a block diagram depicting an example of a computing environment in which a fairness evaluation computing system can be built and applied in a fairness and bias assessment application, according to certain aspects of the present disclosure

[0009] FIG. 2 is a flowchart of a method for implementing a fairness evaluation system for determining lack of fairness of a predictive model according to disclosed embodiments.

[0010] FIG. 3 is a flowchart of a method for implementing a fairness evaluation system for determining lack of fairness of a predictive model including proxies according to disclosed embodiments.

[0011] FIG. 4. is a flowchart of a method for implementing a fairness evaluation system for determining lack of fairness of a predictive model using a unified metric for fairness evaluation according to disclosed embodiments.

[0012] FIG. 5 illustrates a method for implementing a fairness evaluation system for defining a unified metric when from categorical values according to certain embodiments.

[0013] FIG. 6 is a block diagram depicting an example of a computing device, which can be used to implement the fairness evaluation server or the model training server according to disclosed embodiments.Detailed Description

[0014] Certain aspects and examples of the present disclosure relate to systems and methods for fairness and bias evaluation in modelled systems. The systems and methods described herein provide the ability to conduct fairness and bias tests that may be extended to any defined fairness test using customer-supplied data, which eliminates the unintentional introduction of false positive results of disparate impact. Further, the techniques described herein remove the problem of storing consumer demographics on local systems while enabling appropriate tests to be conducted to detect fairness or bias in modelled systems. Moreover, the techniques described herein may be used to adjust model scoring, such as credit scoring, such that a scoring system that is both explainable and fair with respect to disparate impact is achieved.

[0015] In some instances, proxies, also referred to as predictive classifiers, such as Bayesian Improved Surname Geocoding (BISG) and Bayesian Improved First NameSurname Geocoding (BIFSG) have been used by modelers to test for fairness or bias in models. BISG for example predicts whether individuals are members of a protected class based on the last names of the individuals and the frequency of occurrence of those last names in defined geographies. BIFSG similarly predicts whether individuals are members of a protected class, but further based on the individuals’ first names. BISG, BIFSG, and other proxies can introduce flaws due to the inherent inaccuracies of the predictions of such models. Thus, the use of BISG, BIFSG, and other proxies in the fairness and bias testing may risk unintentionally introducing bias rather than mitigating the bias. In other words, use of proxies may unintentionally create disparate impact false positive results that provide an output falsely indicating that a model is unfair.

[0016] Instead of relying on proxies, a ground truth set of self-reported class data may be used as part of the fairness and bias testing. The ground truth set of self-reported class data may provide information about whether the proxy itself, or any other source of derived demographics or protected class data, has value in assessing disparate impact of a model, such as a credit score. Using this information, a modeler is able to assess, account for, and mitigate any detected unfairness in models used by the modeler.

[0017] Further, the modeler may deploy appropriate fairness and bias testing with customer supplied demographics (e.g., self-reported demographic information from the insurance industry). This reduces or otherwise eliminates the burden for the modeler to collect the requisite data or to rely on flawed data predictions for individual-level protected class information.

[0018] To conduct a fairness and bias evaluation, the modeler may perform a fairness through unawareness test. This type of test assumes that if a model is unaware of protected-class attributes of individuals on which decisions are made by the model, then the model will be inherently fair. Once a modeler is able to conduct the fairness through unawareness test, any or all other fairness tests may also definitively be tested in any desired sequence using any regulatory compliant scoring system (e.g., credit scoring such as NeuroDecision™ Technology, Loan Portfolio Management , etc.) or any other model that the modeler can ingest.

[0019] The fairness through unawareness test has two parts. The first part is to build two models. A first model is built that excludes any protected class data as a predictor from which predictive scores, such as credit scores, are generated. The first model is the model under test for fairness or bias. A second model is also built using only a subset of data used to build the first model, where the subset of the data is defined by a single selfreported protected-class attribute. The defining protected-class attribute could be women, for example. Once the first and second models are built, scores from each model are indexed into a definite interval (e.g., 1 to 100) and compared. Showing that the credit score distributions are reasonably similar between the two models is accepted as there not being a disparate impact on the protected class in the scores generated by the first model.

[0020] The second part of the test examines the correlation of every attribute in the model (or every credit attribute available) with the protected-class attribute under test. If the correlations are relatively low, this is accepted as not finding a disparate impact on the protected class in the scores generated by the first model. Jointly, the two negative results are interpreted as no evidence for disparate impact for the first model.

[0021] To assess the risk of false positive disparate impact (FPDI) when using proxies to build the second model, a third model may be built using the proxy to define the protected-class attribute (e.g., using BISG to predict racial information about a user based on the user’s surname and geography). The scores between the first and third models may again be indexed and compared, as above, to show whether disparate impact is present in the first model. Subsequently, the second part of the test evaluates the correlations between the proxy BISG protected-class attribute and every attribute in the model, or every attribute available to the modeler under test. If a difference in scores is observed, which is probable given the issues with proxy accuracy, then the result may be interpreted as evidence of FPDI. This is substantially reinforced if the correlations change systematically. Any other source of protected class attributes, such as commercial demographic data, aggregated demographics, etc. may also be tested and evaluated similarly.

[0022] To provide these services, a modeler anonymously transfers the self-reported protected class data from the data source and correctly computed proxies to an alternate site. The modeler subsequently locates the individuals identified in the protected class data in a data file of the individuals lacking the protected class data. Further, the modeler determines a model (generic default, etc.) and uses various methods to build the first and second model variants, where the second model is a subset of the first model. Additionally, the modeler computes correlations between the protected class status attribute and the attributes in the scoring model or all attributes available for the scoring model (e.g., a credit score model).

[0023] The second model may then be rebuilt using proxies to identify the protected class of consumers in the data for use in the subset. Score distributions between those from the first model and the rebuilt second model can be compared, and the correlations between the proxy-predicted protected class status attribute and the attributes available to or used in generating the second model scores are computed. FPDI is interpreted from this comparison as a quantitative distinction between the two sets of results relevant to the first model. That is, FPDI is identified for the rebuilt second model (built with the proxy) but not for the initial second model. This indicates that FPDI is present when using proxy to identify protected classes.

[0024] In other words, testing proxy predictive classifiers, such as BISG and BIFSG, for FPDI may be performed by receiving self-reported data from the customer, matching the self-reported data to a data file without protected class information, building three credit scoring models, computing two sets of correlations, and quantifying the differences between the scores and correlations. Thus, the modeler is able to quantify the differences between results obtained using ground-truth, self-reported protected class data and results obtained using any other measures that represent protected class data.

[0025] Certain aspects described herein overcome the limitations of previous techniques and provide improvements to modeled systems (e.g., machine learning model systems) and their corresponding database by preventing the transmission and storage of faulty, biased data generated by the modeled system. The system can provide fullautomation of model performance monitoring and detection of biases in such models. Detection of disparate impact and false positive disparate impact as crossing respective thresholds can cause the described systems to identify a model as fundamentally inoperative and harmful, and thereby automatically flag such models as unfair and further prevent data generated by the inoperative systems from being stored in external databases.

[0026] Illustrative examples are given to introduce the reader to the general subject matter discussed herein and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional features and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative aspects, but, like the illustrative aspects, should not be used to limit the present disclosure.Operating Environment Example for Fairness Evaluation Operations

[0027] Referring now to the drawings, FIG. 1 is a block diagram depicting an example of a computing environment 100 in which a fairness evaluation computing system 130 can be built and applied in a fairness and bias assessment application. FIG. 1 depicts examples of hardware components of a fairness evaluation computing system 130 according to some aspects. The fairness evaluation computing system 130 can be a specialized computing system that may be used for processing large amounts of data using a large number of computing processing cycles. The fairness evaluation computing system 130 can include a network training server 110 for building multiple score prediction models wherein the score prediction models may be built from various training samples 124 in a data repository 122. Each of the score prediction models can include an input layer, one or more hidden layers, and an output layer. Each layer contains one or more nodes connected to the nodes in another layer. The fairness evaluation computing system 130 can include a fairness evaluation server 118 for performing fairness evaluation for agiven score prediction model 130 using the a combination of the score prediction models 120.

[0028] The network training server 110 can include one or more processing devices that execute program code, such as a network training application 112. The program code can be stored on a non-transitory computer-readable medium. The network training application 112 can execute one or more processes to train and optimize a neural network or other score prediction model 120 for predicting risk indicators based on training samples 124 and / or proxies 126.

[0029] In some aspects, the network training application 112 can build and train a score prediction models 120 utilizing training samples 124 and proxies 126. Each score prediction model can be trained with training samples 126. Other score prediction models may be trained with the training samples 124 in addition to the proxies 126. The training samples 124 can include multiple training vectors consisting of training data for predictor variables and training score outputs corresponding to the training vectors. The training samples can be stored in one or more network attached storage units on which various repositories, databases, or other structures are stored. Examples of these data structures are the risk data repository 122.

[0030] Network-attached storage units may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, the network-attached storage unit may include storage other than primary storage located within the network training server 110 that is directly accessible by processors located therein. In some aspects, the network-attached storage unit may include secondary, tertiary, or auxiliary storage, such as large hard drives, servers, virtual memory, among other types. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be stored and that does not include carrier waves or transitory electronic signals. Examples of a non-transitorymedium may include, for example, a magnetic disk or tape, optical storage media such as a compact disk or digital versatile disk, flash memory, memory, or memory devices.

[0031] The fairness evaluation server 118 can include one or more processing devices that execute program code, such as a fairness evaluation application 114. The program code is stored on a non-transitory computer-readable medium. The fairness evaluation application 114 can execute one or more processes to utilize the score prediction models 120 trained by the model training application 112 based on input predictor variables . In addition, the score predictions models 120 can also be utilized to generate explanator data for the proxies 126 and training samples 124 which can indicate relationships between different score prediction models 120 and corresponding inputs.

[0032] Furthermore, the fairness evaluation computing system 130 can communicate with various other computing systems, such as client computing systems 104. For example, client computing systems 104 may send fairness assessment queries to the fairness evaluation server 118 for fairness assessment of a given model or may send signals to the fairness assessment server 118 that control or otherwise influence different aspects of the fairness assessment computing system 130. The client computing systems 104 may interact with the fairness evaluation sever 118 via public data network 108.

[0033] Each client computing system 104 may include one or more third-party devices, such as individual servers or groups of servers operating in a distributed manner. A client computing system 104 can include any computing device or group of computing devices operated by a seller, lender, or other providers of products or services. The client computing system 104 can include one or more server devices. The one or more server devices can include or can otherwise access one or more non-transitory computer- readable media. The client computing system 104 can also execute instructions that provide an interactive computing environment accessible to user computing systems 106. Examples of the interactive computing environment include a mobile application specific to a particular client computing system 104, a web-based application accessible via a mobile device, etc. The executable instructions are stored in one or more non-transitory computer-readable media.

[0034] The client computing system 104 can further include one or more processing devices that are capable of providing the interactive computing environment to perform operations described herein. The interactive computing environment can include executable instructions stored in one or more non-transitory computer-readable media. The instructions providing the interactive computing environment can configure one or more processing devices to perform operations described herein. In some aspects, the executable instructions for the interactive computing environment can include instructions that provide one or more graphical interfaces. The graphical interfaces are used by a user computing system 106 to access various functions of the interactive computing environment. For instance, the interactive computing environment may transmit data to and receive data from a user computing system 106 to shift between different states of the interactive computing environment, where the different states allow one or more electronics transactions between the user computing system 106 and the client computing system 104 to be performed.

[0035] In some examples, a client computing system 104 may have other computing resources associated therewith (not shown in FIG. 1), such as server computers hosting and managing virtual machine instances for providing cloud computing services, server computers hosting and managing online storage resources for users, server computers for providing database services, and others. The interaction between the user computing system 106 and the client computing system 104 may be performed through graphical user interfaces presented by the client computing system 104 to the user computing system 106, or through an application programming interface (API) calls or web service calls.

[0036] The user computing system 106 can include any computing device or other communication device operated by a user, such as a consumer or a customer. The user computing system 106 can include one or more computing devices, such as laptops, smartphones, and other personal computing devices. A user computing system 106 can include executable instructions stored in one or more non-transitory computer-readable media. The user computing system 106 can also include one or more processing devicesthat are capable of executing program code to perform operations described herein. In various examples, the user computing system 106 can allow a user to access certain online services from a client computing system 104 or other computing resources, to engage in mobile commerce with a client computing system 104, to obtain controlled access to electronic content hosted by the client computing system 104, etc.

[0037] For instance, the user can use the user computing system 106 to engage in one or more electronic transactions with a client computing system 104 via an interactive computing environment. The one or more transactions can be determined based on the set of steps output by the fairness evaluation computing system 130 in response to receiving a model fairness query from the user computing system 106. In some examples, the fairness evaluation computing system 130 can receive training samples 124 from the user computing system 106.

[0038] In a simplified example, the system depicted in FIG. 1 can configure multiple score prediction models to be used both for determining risk indicators, such as credit scores, using predictor variables, and further evaluating the fairness of the given score prediction model 120. A first model of the score prediction models may be trained on ground truth data, or an otherwise unbiased data set, while subsequent score prediction models 120 may be trained on attributes indexed by protected class attributes (e.g., gathered as self-reported data), or proxies 126 which may represent a confidence that a protected class attribute is present. Determining correlations in scores generated by the first model, and the subsequent models may be used to determine which of score prediction models 120 is unfair or otherwise biased with respect to the protected class attributes.

[0039] Examples of protected class attributes, which may be gathered by self-report data, or recreated by proxies include, but are not limited to, attributes identifying race, gender, family and the like. Examples of proxies used to recreate the protected class attributes include but are not limited to BISG and BIFSG as discussed above.

[0040] Each communication within the operating environment 100 may occur over one or more data networks, such as a public data network 108, a network 116 such as a private data network, or some combination thereof. A data network may include one or more of a variety of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (“LAN”), a wide area network (“WAN”), or a wireless local area network (“WLAN”). A wireless network may include a wireless interface or a combination of wireless interfaces. A wired network may include a wired interface. The wired or wireless networks may be implemented using routers, access points, bridges, gateways, or the like, to connect devices in the data network.

[0041] The number of devices depicted in FIG. 1 is provided for illustrative purposes. Different numbers of devices may be used. For example, while certain devices or systems are shown as single devices in FIG. 1 , multiple devices may instead be used to implement these devices or systems. Similarly, devices or systems that are shown as separate, such as the model training server 110 and the fairness evaluation server 118, may be instead implemented in a signal device or system.Examples of Operations Involving Model Bias Prediction

[0042] FIG. 2 is a flowchart of a process 200 for implementing a fairness evaluation system for determining the fairness, or lack thereof, of a predictive model according to disclosed embodiments. One or more computing devices (e.g., the fairness evaluation server 118) implement operations depicted in FIG. 2 by executing suitable program code (e.g., the fairness evaluation application 114). For illustrative purposes, the process 200 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible. While the blocks of the process 200 aredescribed in the temporal order below for illustrative purposes, it may be appreciated that the blocks can occur in any order and some blocks may occur simultaneously.

[0043] At block 202, the process 200 involves building a first model including a first data set, wherein the first data set excludes protected class data as a predictor value. According to more direct tests, the first model would generally be the model under test. However, the first model, according to aspects of the disclosure, may also be used to test other models to determine the other models for lack of fairness. As the first model excludes protected class data (e.g., race, sex, gender) in addition to excluding proxies (e.g., BISG or BIFSG data), the first model, according to certain examples (e.g., when the first model does not generate indications of disparate impact) may provide a control environment for testing of fairness according to other models.

[0044] At block 204, the process 200 involves building a second model including a second data set, wherein the second data set includes a subset of the first data set, and wherein the second data set is further defined by one self-reported protected class attribute. Examples of self-reported protected class attributes include binary attributes (e.g., sex), continuous attributes (e.g., age), or category based attributes (e.g., race). According to other examples, such as those discussed with respect to FIG. 4, multiple self-reported class attributes, including different types of class attributes may define the second data set. The self-reported protected attributes, defining the second data set, provide a ground-truth which may be used indicate lack of fairness of the first model and / or other models such as a third model including proxies as discussed with respect to FIG. 3.

[0045] At block 206, the process 200 involves generating and indexing a first score distribution from the first model and generating and indexing a second score distribution from the second model. The first and second score distributions may be indexed into a definite interval (e.g., an interval [1 ,100]). The first and second score distributions may beindexed into the same or separate data structures but will generally have the same index to augment comparison of the first and second score distributions.

[0046] At block 208, the process 200 involves comparing the first score distribution against the second score distribution to determine a self-report correlation. The self-report correlation, also referred to as the ground-truth correlation, represents the degree of similarity of outputs from the first and second score distributions as output by their respective models. Various metrics may be used to define the strength of the self-report correlation. For example, one self-report correlation may comprise a value from 0 to 1 , wherein a score of 0 represents no indication of correlation (i.e. , no degree of similarity), and 1 represents absolute correlation. In other examples, self-report correlations may be truncated to binary or represented by other data values. High self-report correlation values, indicating a high degree of similarity between the outputs of the first model, and the second model, where the second model is explicitly trained on a subset of data indexed by a self-reported protected attribute, may thus be indicative of a finding of no disparate impact or lack of fairness in the first model.

[0047] At block 210, the process 200 involves comparing the self-report correlation against a first disparate impact threshold, and in response to determining the self-report correlation is less than the first disparate impact threshold, generating a first prediction of disparate impact. The first disparate impact threshold may be a configurable value representing a tolerance with respect to potential lack of fairness detected by the selfreport correlation. For instance, the disparate impact threshold may be a value of .8, where self-report correlations less than a value of .8, indicates a weaker correlation between the first model and the second, ground truth self-report model. In other words, determining that the first score and the second score, generated by their respective models, are reasonably similar by surpassing the first disparate impact threshold may be accepted as a finding of no disparate impact or lack of fairness in the first model, whilefailing to exceed the first disparate impact indicates insufficient similarity between the first model’s and second model’s scores and an indicia of disparate impact.

[0048] At block 212, the process 200 involves determining an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, wherein the plurality of attributes are generated by the first model. In other words, the process examines the correlation of every attribute in the first model, or every credit attribute available, with the protected class attribute under test.

[0049] At block 214, the process 200 involves comparing the aggregate correlation against a second disparate impact threshold. The second disparate impact threshold, like the first disparate impact threshold of block 210, may be a configurable value representing a tolerance, but with respect to the degree of similarity between every attribute of the model and the protected class attribute under test. Exceeding the second disparate impact threshold may provide evidence of disparate impact or lack of fairness. Thus, the first and second disparate impact thresholds provide means for evidencing disparate impact and lack of fairness.

[0050] In some examples, negative results from each comparison at blocks 210 and 214, may be used to provide evidence for no disparate impact or lack of fairness. Evidence of disparate impact may also be used to configure external databases and prevent such databases from receiving faulty (e.g., biased) data. For example, determining that the self-report correlation is under the first disparate impact threshold, and that the aggregate correlation exceeds the second disparate impact threshold, can cause the fairness evaluation computing system 130 to both generate an alert indicating the first model produces a disparate impact in generating prediction scores, and further preventing storage of data generated by the first model in an external database.Examples of Operations Involving Model Fairness Evaluation of Proxy Data

[0051] As discussed above with respect to FIG. 2, certain examples provide the ability to perform fairness through unawareness testing where a first data model is compared against a second ground truth data model, and correlations extracted from thecomparison. In such a way, the fairness of the first model can be computed. According to additional examples, other models of fairness tests may be evaluated for lack of fairness. For instance, the example process described above may be replicated for each proxy by replacing the second model’s data set with a proxy data set to define the protected class attribute. FIG. 3 is a flowchart of a method for implementing a fairness evaluation system for determining the fairness of a predictive model including proxies according to disclosed embodiments.

[0052] At block 302, the process 300 involves building a third model data set, wherein the third data set includes a proxy predicting a protected class attribute. Thus instead of using ground truth, self-reported protected class attributes, the third model can be built and trained on proxies such as BISG or BIFSG data.

[0053] At block 304, the process 300 involves generating and indexing a third score distribution from the third model. Assuming aspects of process 200 have already been, performed (e.g., generating the first model based on the first data set, and generating the first score distribution from the first model), the third score distribution can be indexed and compared against the first score distribution.

[0054] At block 306, the process 300 involves comparing the first score distribution against the third score distribution to determine a proxy correlation. Block 306 is similar to block 208 of process 200 but iterated with the third model including the proxy data.

[0055] At block 308, the process 300 involves comparing the proxy correlation against the first disparate impact threshold and in response to determining the proxy correlation exceeds the first disparate impact threshold, generating a first prediction of false positive disparate impact. Block 308 is similar to block 210 of process 200, however, by comparing the proxy correlation against the threshold, process 300 explicitly tests the third model for indications of false positive disparate impact, which are indicators of fairness of the proxy being incorporated into model training sets.

[0056] At block 310, the process 300 involves determining a second aggregate correlation for a plurality of attributes in the first model with the proxy, wherein the plurality of attributes are generated by the first model.

[0057] At block 312, the process 300 involves comparing the second aggregate correlation against the second disparate impact threshold, and in response to determining the second aggregate correlation exceeds the second disparate impact threshold, generating a second prediction of false positive disparate impact. Blocks 310-312 are similar to blocks 212-214 of process 200 but iterate the correlation testing to evaluate correlations between the proxy protected class attribute and the credit attributes generated by the first and third models.Examples of Operations For Generating a Unified Metric for Fairness Evaluation

[0058] In some instances, particularly in multivariate cases where fairness testing includes determining correlations between dissimilar attribute and score types (e.g., wherein the model within the test is multivariate and defined by multiple self-reported protected class attributes of different types such as binary attribute and a category based attribute, or wherein the model output is continuous value such as a score, while the protected class attribute is a binary attribute) a unified metric for assessing correlations may augment the process of detecting lack of fairness. The multivariate case, enabled by the unified metric, could consider for instance, testing a model which predicts home loan approval (a binary yes / no) as the target output in view of protected attributes including age (a continuous value), sex (a binary value) and race (a categorical value) during simultaneous testing.

[0059] FIG. 4 is a flowchart of a method for implementing a fairness evaluation system for determining the fairness of a predictive model using a unified metric for fairness measurement according to disclosed embodiments.

[0060] At block 402, the process 400 receives two or more self-reported protected class attributes. The class attributes can be of the same or different attribute types. For instance, a first self-reported protected class attribute can be binary type such as militarystatus and a second-self-reported protected class attribute can be a continuous type such as age.

[0061] At decision 404, the process 400 determines whether at least one of the two or more self-reported protected class attributes includes one or more categorical values. Categorical values, unlike binary and continuous values, fail to provide a direct comparison between values within a set. To provide the ability to compare values within the set (e.g., through the universal metric) the process 400, once identifying a categorical value, further processes the categorical value and converts the categorical value into an aggregate categorical score per block 406. If each value identified is not a categorical value (e.g., each are either binary or continuous) no additional conversion may be necessary before proceeding to block 408.

[0062] At block 406, in response to identifying at least one of the values is a categorical value, the process 400 includes, for each identified categorical value, converting the categorical value into an aggregate categorical score. FIG. 5 illustrates an example of the conversion process. Generally, the conversion process includes one-hot encoding the categorical values and performing a pairwise comparison of each category.

[0063] At block 408, the process 400 includes concatenating each of the self-reported protected class attributes together. As in block 406, if a categorical value is identified in the self-reported protected class attributes, the process 400 can include first converting the categorical value into an aggregate score prior to concatenating the aggregate categorical score with each of the remaining self-reported protected class attributes.

[0064] At block 410, the process 400 includes determining a unified metric as a component of the aggregate correlation. In one example (as shown in FIG. 5), the unified metric can comprise a Kendall’s Tau coefficient. In other examples, different unified metrics can be used such as Pearson correlations, Spearman associations and the like. More broadly, the unified metric may refer to any operation used to determine a correlation between multi-variate sets of data. The aggregate correlation can be for instance, the first aggregate correlation of FIG. 2 where the plurality of attributesgenerated by a model under test is compared against multiple self-reported protected class attributes; additionally or alternatively, the aggregate correlation determined by process 400 can be the aggregate correlation for a plurality. In such a way, the process 400 can be integrated with the processes 200, 300 to provide a more refined way of testing for fairness by considering multivariate cases or cases wherein mismatching attributes are to be compared.

[0065] FIG. 5 illustrates a method for implementing a fairness evaluation system for defining a unified metric when from categorical values according to certain embodiments. Data set 502 shows a continuous variables under column C compared against a set of categorical, or nominal values A-C of column N. Each column C and N can represent a target score output of the prediction models or variable under test such as the selfreported protected class data or the proxy.

[0066] Data set 504 shows the categorical values of the data set 502 converted to a one-hot encoding embedding. From data set 504, pairwise comparisons 506a-506c for each categorical value may be generated. By providing pairwise comparisons for each categorical value, a weighted score for each pairwise categorical score may be generated. For instance, in the pairwise comparison of FIG. 5, a unified metric comprising a Kendall’s Tau coefficient is determined for each pairwise comparison. Other unified metrics may be used according to other examples.

[0067] Once each pairwise categorical score is generated, the pairwise categorical scores may be aggregated to determine an aggregate categorical score (e.g., aggregate categorical score 508). Aggregation may comprise, weighted means as shown, taking the ratio of the min / max values, or other the like. The process described above may be iterated for various combinations of targets including scores predicted by first, second, and third models, in addition to the self-reported protected class data and / or proxy data.Example of Computing System for Machine-Learning Operations

[0068] Any suitable computing system or group of computing systems can be used to perform the operations described herein. For example, FIG. 6 is a block diagramillustrating an example of a computing device 600, which can be used to implement the fairness and fairness testing described above. The computing device 600 can include various devices for communicating with other devices in a computing environment.

[0069] The computing device 600 can include a processor 602 that is communicatively coupled to a memory 604. The processor 602 can execute computer-executable program code stored in the memory 604, can access information stored in the memory 604, or both. Program code may include machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, among others.

[0070] Examples of a processor 602 can include a microprocessor, an applicationspecific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other suitable processing device. The processor 602 can include any suitable number of processing devices, including one. The processor 602 can include or communicate with a memory 604. The memory 604 can store program code that, when executed by the processor 602, causes the processor 602 to perform the operations described herein.

[0071] The memory 604 can include any suitable non-transitory computer-readable medium. The computer-readable medium can include any electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable program code or other program code. Non-limiting examples of a computer-readable medium can include a magnetic disk, memory chip, optical storage, flash memory, storage class memory, ROM, RAM, an ASIC, magnetic storage, or any other medium from which a computer processor can read and execute program code. The program code may include processor-specific program code generated by a compiler or an interpreter from code written in any suitable computer-programming language. Examples of suitableprogramming language can include Hadoop, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, ActionScript, etc.

[0072] The computing device 600 may also include a number of external or internal devices such as input or output devices. For example, the computing device 600 is illustrated with an input / output interface 608 that can receive input from input devices or provide output to output devices. A bus 606 can also be included in the computing device 600. The bus 606 can communicatively couple one or more components of the computing device 600.

[0073] The computing device 600 can execute program code 614 that can include the presently described models, testing operations, and the like. The program code 614 may be resident in any suitable computer-readable medium and may be executed on any suitable processing device. Executing the program code 614 can configure the processor 602 to perform one or more of the operations described herein.

[0074] In some aspects, the computing device 600 can include one or more output devices. One example of an output device can be the network interface device 610 depicted in FIG. 6. A network interface device 610 can include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks described herein. Non-limiting examples of the network interface device 610 can include an Ethernet network adapter, a modem, etc.

[0075] Another example of an output device can include the presentation device 612 depicted in FIG. 6. A presentation device 612 can include any device or group of devices suitable for providing visual, auditory, or other suitable sensory output. Non-limiting examples of the presentation device 612 can include a touchscreen, a monitor, a speaker, a separate mobile computing device, etc. In some aspects, the presentation device 612 can include a remote client-computing device that can communicate with the computing device 600 using one or more data networks described herein. In other aspects, the presentation device 612 can be optional.

[0076] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure.General Considerations

[0077] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these specific details. In other instances, methods, apparatuses, or systems that would be known by one of ordinary skill have not been described in detail so as not to obscure claimed subject matter.

[0078] Unless specifically stated otherwise, it is appreciated that throughout this specification that terms such as “processing,” “computing,” “determining,” and “identifying” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0079] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computing systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more aspects of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may beused to implement the teachings contained herein in software to be used in programming or configuring a computing device.

[0080] Aspects of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied — for example, blocks can be re-ordered, combined, or broken into sub-blocks. Certain blocks or processes can be performed in parallel.

[0081] The use of “adapted to” or “configured to” herein is meant as open and inclusive language that does not foreclose devices adapted to or configured to perform additional tasks or steps. Additionally, the use of “based on” is meant to be open and inclusive, in that a process, step, calculation, or other action “based on” one or more recited conditions or values may, in practice, be based on additional conditions or values beyond those recited. Headings, lists, and numbering included herein are for ease of explanation only and are not meant to be limiting.

[0082] While the present subject matter has been described in detail with respect to specific aspects thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily produce alterations to, variations of, and equivalents to such aspects. Any aspects or examples may be combined with any other aspects or examples. Accordingly, it should be understood that the present disclosure has been presented for purposes of example rather than limitation, and does not preclude inclusion of such modifications, variations, or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art.

Claims

Claims1. A method that includes one or more processing devices performing operations comprising: building a first model including a first data set, wherein the first data set excludes protected class data as a predictor value; building a second model including a second data set, wherein the second data set includes a subset of the first data set and wherein the second data set is further defined by one self-reported protected class attribute; generating and indexing a first score distribution from the first model; generating and indexing a second score distribution from the second model; comparing the first score distribution against the second score distribution to determine a self-report correlation; comparing the self-report correlation against a first disparate impact threshold; in response to determining the self-report correlation exceeds the first disparate impact threshold, generating a first prediction of disparate impact; determining an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, wherein the plurality of attributes being generated by the first model; comparing the aggregate correlation against a second disparate impact threshold; and in response to determining the aggregate correlation exceeds the second disparate impact threshold, generating a second prediction of disparate impact.

2. The method of claim 1 further comprising: building a third model including a third data set, wherein the third data set includes a proxy defining a protected-class attribute; generating and indexing a third score distribution from the third model; comparing the first score distribution against the third score distribution to determine a proxy correlation;comparing the proxy correlation against the first disparate impact threshold; and in response to determining the proxy correlation exceeds the first disparate impact threshold, generating a first prediction of false positive disparate impact.

3. The method of claim 2, wherein the proxy comprises Bayesian Improved Surname Geocoding (BISG) data.

4. The method of claim 2, wherein the proxy comprises Bayesian Improved First Name Surname Geocoding (BIFSG) data.

5. The method of claim 1 , wherein in response to both determining that the self-report correlation exceeds the first disparate impact threshold, and that the aggregate correlation exceeds the second disparate impact threshold, generating and displaying an alert indicating the first model produces a disparate impact, and further preventing storage of data generated by the first model in an external database.

6. The method of claim 1 , wherein the second data set is further defined by multiple self-reported protected class attributes; wherein determining the self-report correlation comprises: encoding and concatenating each of the self-reported protected class attributes together; and determining a unified metric as a component of the self-report correlation.

7. The method of claim 6 wherein at least one of the multiple self-reported protected class attributes comprises a categorical value, and for each categorical value, converting the categorical value into an aggregate categorical score prior to concatenating the aggregate categorical score with each of the remaining self-reported protected class attributes.

8. A system comprising:a processing device; and a memory device in which instructions executable by the processing device are stored for causing the processing device to perform operations comprising: building a first model including a first data set, wherein the first data set excludes protected class data as a predictor value; building a second model including a second data set, wherein the second data set includes a subset of the first data set and wherein the second data set is further defined by one self-reported protected class attribute; generating and indexing a first score distribution from the first model; generating and indexing a second score distribution from the second model; comparing the first score distribution against the second score distribution to determine a self-report correlation; comparing the self-report correlation against a first disparate impact threshold; in response to determining the self-report correlation exceeds the first disparate impact threshold, generating a first prediction of disparate impact; determining an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, wherein the plurality of attributes being generated by the first model; and comparing the aggregate correlation against a second disparate impact threshold; and in response to determining the aggregate correlation exceeds the second disparate impact threshold, generating a second prediction of disparate impact.

9. The system of claim 8, wherein the operations further comprise: building a third model including a third data set, wherein the third data set includes a proxy defining a protected-class attribute;generating and indexing a third score distribution from the third model; comparing the first score distribution against the third score distribution to determine a proxy correlation; and comparing the proxy correlation against the first disparate impact threshold; and in response to determining the proxy correlation exceeds the first disparate impact threshold, generating a first prediction of false positive disparate impact.

10. The system of claim 9 wherein the proxy comprises Bayesian Improved Surname Geocoding (BISG) data.11 . The system of claim 9 wherein the proxy comprises Bayesian Improved First Name Surname Geocoding (BIFSG) data.

12. The system of claim 8 wherein in response to both determining that the self-report correlation exceeds the first disparate impact threshold, and that the aggregate correlation exceeds the second disparate impact threshold, generating and displaying an alert indicating the first model produces a disparate impact, and further preventing storage of data generated by the first model in an external database.

13. The system of claim 8 wherein the second data set is further defined by multiple self-reported protected class attributes; wherein determining the self-report correlation comprises: encoding and concatenating each of the self-reported protected class attributes together; and determining a unified metric as a component of the self-report correlation.

14. The system of claim 13 wherein at least one of the multiple self-reported protected class attributes comprises a categorical value, and for each categorical value, converting the categorical value into an aggregate categorical score prior to concatenating theaggregate categorical score with each of the remaining self-reported protected class attributes.

15. A non-transitory computer-readable storage medium having program code that is executable by a processor to cause a computing device to perform operations, the operations comprising: building a first model including a first data set, wherein the first data set excludes protected class data as a predictor value; building a second model including a second data set, wherein the second data set includes a subset of the first data set and wherein the second data set is further defined by one self-reported protected class attribute; generating and indexing a first score distribution from the first model; generating and indexing a second score distribution from the second model; comparing the first score distribution against the second score distribution to determine a self-report correlation; comparing the self-report correlation against a first disparate impact threshold; in response to determining the self-report correlation exceeds the first disparate impact threshold, generating a first prediction of disparate impact; determining an aggregate correlation for a plurality of attributes in the first model with the self-reported protected class attribute, wherein the plurality of attributes being generated by the first model; and comparing the aggregate correlation against a second disparate impact threshold; and in response to determining the aggregate correlation exceeds the second disparate impact threshold, generating a second prediction of disparate impact.

16. The non-transitory computer-readable storage medium of claim 15, wherein the operations further comprise:building a third model including a third data set, wherein the third data set includes a proxy defining a protected-class attribute; generating and indexing a third score distribution from the third model; comparing the first score distribution against the third score distribution to determine a proxy correlation; and comparing the proxy correlation against the first disparate impact threshold; and in response to determining the proxy correlation exceeds the first disparate impact threshold, generating a first prediction of false positive disparate impact.

17. The non-transitory computer-readable storage medium of claim 16, wherein the proxy comprises Bayesian Improved Surname Geocoding (BISG) data.

18. The non-transitory computer-readable storage medium of claim 16, wherein the proxy comprises Bayesian Improved First Name Surname Geocoding (BIFSG) data.

19. The non-transitory computer-readable storage medium of claim 15, wherein in response to both determining that the self-report correlation exceeds the first disparate impact threshold, and that the aggregate correlation exceeds the second disparate impact threshold, generating and displaying an alert indicating the first model produces a disparate impact, and further preventing storage of data generated by the first model in an external database.

20. The non-transitory computer-readable storage medium of claim 15, wherein the second data set is further defined by multiple self-reported protected class attributes; wherein determining the self-report correlation comprises: encoding and concatenating each of the self-reported protected class attributes together; and determining a unified metric as a component of the self-report correlation.

Images