Method and system for detecting and monitoring bias in software applications

Abstract

A method, system, for detecting and monitoring bias in an application includes indexing training data, obtaining multiple correlation values ​​between one or more features in the indexed training data and a target variable, calculating a first value and a favorable outcome for each of the one or more features, and a second value along with an unfavorable outcome, calculating an absolute value of the difference between the calculated first value and the calculated second value, and calculating a sum of the calculated absolute values ​​of the multiple correlation values ​​for one of the one or more features.

Description

Technical Field

[0001] This application generally relates to managing software applications, and more particularly to detecting and monitoring bias in software applications using artificial intelligence and to the device thereof.

Background Art

[0002] Data of different types or formats are generally used by humans or computers to make decisions. For example, a bank may base a decision on whether to extend a certain total amount of credit to a person or business based on credit scores, past credit history, income, and other demographic information. Unfortunately, such decisions may contain some form of bias (e.g., implicit bias), and thus, even if an institution believes its decision-making process to be fair, in reality, the results may be biased. Bias refers to making a decision using a decision-making process that favors or does not favor certain traits or characteristics such as gender, age, ethnic background, geographical location, or income. As a result, bias may creep into the rules used to approve or reject applications such as credit applications, and this bias can be difficult to detect. Existing technologies attempt to solve the technical problem of detecting bias in computer-made decisions by requiring employees such as data scientists to manually review data sets, match rules, and correlate data and rules to the decisions that detect bias. Still, the techniques used by existing technologies require manual intervention, which leads to errors, and thus do not completely solve the technical problem. Moreover, it is very difficult for the user himself to find out which features may be biased.

[0003] To solve the above problems, it is desirable to provide a new system and method for detecting and monitoring bias in an application.

Summary of the Invention

[0004] Embodiments provide a computer-executable method for monitoring and detecting bias in an application, including indexing training data and obtaining a plurality of correlation values between one or more features and a target variable in the indexed training data. For each of the one or more features, a first value and a favorable result, as well as a second value and an unfavorable result, are calculated. The absolute value of the difference between the calculated first value and the calculated second value is calculated. The sum of the calculated absolute values of the plurality of correlation values for one of the one or more features is calculated.

[0005] Embodiments further provide a computer-executable method that further includes indexing feedback data received from a client device.

[0006] Embodiments further provide a computer-executable method that further includes obtaining training data from a training data server.

[0007] Embodiments further provide a computer-executable method that further includes calculating a second sum of the calculated absolute values of the plurality of correlation values for one of the one or more features for the feedback data.

[0008] Embodiments further provide a computer-executable method that further includes presenting a correlation change value determined by the difference between the calculated sum of the calculated absolute values associated with the training data and the calculated sum of the calculated absolute values associated with the feedback data.

[0009] The embodiments further provide a computer-executable method in which a correlation change value represents a candidate feature to be monitored.

[0010] The embodiments further provide a non-transitory computer-readable medium including indexing training data and obtaining a plurality of correlation values between one or more features and a target variable in the indexed training data. For each of the one or more features, a first value and a favorable outcome, as well as a second value and an unfavorable outcome, are calculated. The absolute value of the difference between the calculated first value and the calculated second value is calculated. The sum total of the calculated absolute values of the plurality of correlation values for one of the one or more features is calculated.

[0011] In another illustrative embodiment, a non-transitory computer-readable medium is provided that comprises a computer-usable or readable medium having a computer-readable program. The computer-readable program, when executed by a processor, causes the processor to perform various ones and combinations of the operations outlined above in connection with the illustrative embodiments of the method.

[0012] In yet another illustrative embodiment, a system is provided. The system can comprise a full-question generation processor configured to perform various ones and combinations of the operations outlined above in connection with the illustrative embodiments of the method.

[0013] Additional features and advantages of the present disclosure will become apparent from the following detailed description of illustrative embodiments, which proceeds with reference to the accompanying drawings.

[0014] The foregoing and other aspects of the present invention will be best understood from the following detailed description when read in conjunction with the accompanying drawings. In order to illustrate the present invention, presently preferred embodiments are shown in the drawings, it being understood, however, that the invention is not limited to the particular means disclosed. The following figures are included in the drawings.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 6C

DETAILED DESCRIPTION OF THE INVENTION

[0016] The present invention may be a system, method, or computer program product, or a combination thereof, for detecting and monitoring bias in a software application using artificial intelligence (AI). The computer program product may include a computer-readable storage medium (or media) including computer-readable program instructions for causing a processor to execute aspects of the present invention.

[0017] A network environment 10 including an example of a bias monitoring computing system 14 is shown in FIGS. 1-2. In this particular example, the environment 10 includes a bias monitoring computing system 14, one or more client devices 12(1)-12(n), one or more training data servers 16(1)-16(n), and one or more application servers 17(1)-17(n) connected via one or more communication networks 30. However, the environment can include other types and numbers of systems, devices, components, or other elements, or combinations thereof, that are well known in the art and not shown or described herein. This technology provides several advantages including a method, non-transitory computer-readable medium, and system for detecting and monitoring bias in software applications using artificial intelligence (AI).

[0018] Referring more particularly to FIGS. 1-2, the bias monitoring computing system 14 is programmed to detect and monitor bias in an application using artificial intelligence. Referring now to FIG. 2, the bias monitoring computing system 14 can employ a hub architecture including a north bridge and memory controller hub (NB / MCH) 201 and a south bridge and input / output (I / O) controller hub (SB / ICH) 202. A processing unit 203, main memory 204, and graphics processor 205 can be connected to the NB / MCH 201. The graphics processor 205 can be connected to the NB / MCH 201 through an accelerated graphics port (AGP).

[0019] In the example of the description, the network adapter 206 is connected to the SB / ICH 202. The audio adapter 207, the keyboard and mouse adapter 208, the modem 209, the read-only memory (ROM) 210, the hard disk drive (HDD) 211, the optical drive (CD or DVD) 212, the universal serial bus (USB) ports and other communication ports 213, and the PCI / PCIe devices 214 can be connected to the SB / ICH 702 through the bus system 216. The PCI / PCIe devices 214 can include an Ethernet (R) adapter, an add-in card, and a PC card for a notebook computer. The ROM 210 can be, for example, a flash basic input / output system (BIOS). The HDD 211 and the optical drive 212 can use an integrated drive electronics (IDE) or a serial advanced technology attachment (SATA) interface. The super I / O (SIO) device 215 can be connected to the SB / ICH.

[0020] The operating system can be executed on the processing unit 203. The operating system can cooperate to provide control of various components within the bias monitoring computing system 14. The operating system as a client can be a commercially available operating system. An object-oriented programming system such as the Java(R)(TM) programming system can execute with the operating system and provide calls to the operating system from an object-oriented program or application executed on the data processing system 700. The bias monitoring computing system 14 as a server can be an IBM(R) eServer(TM) system p(R) executing an Advanced Interactive Executive operating system or a Linux operating system. The bias monitoring computing system 14 can be a symmetric multi-processor (SMP) system that can include multiple processors in the processing unit 203. Alternatively, a single processor system may be employed.

[0021] Instructions for the operating system, object-oriented programming system, and application or program are placed on a storage device such as the HDD 211 and loaded into the main memory 204 for execution by the processing unit 203. The process for an embodiment of the full question generation system can be implemented by the processing unit 703 using computer-usable program code, and the computer-usable program code can be placed in a memory such as the main memory 204, ROM 210, etc., or in one or more peripheral devices.

[0022] The bus system 216 can be composed of one or more buses. The bus system 216 can be implemented using any type of communication fabric or architecture that can be provided for data transfer between different components or devices attached to the communication fabric or architecture. A communication unit such as the modem 209 or the network adapter 206 can include one or more devices that can be used to transmit and receive data.

[0023] Those skilled in the art will understand that the hardware depicted in FIG. 2 may vary depending on the implementation form. For example, the bias monitoring computing system 14 includes some components that should not be directly included in some of the embodiments shown in FIGS. 3 to 6C. Nevertheless, it should be understood that the embodiments shown in FIGS. 3 to 6C can include one or more of the components and configurations of the bias monitoring computing system 14 for implementing the processing methods and steps according to the disclosed embodiments.

[0024] Moreover, other internal hardware or peripheral devices such as flash memory, equivalent non-volatile memory, or optical disk drives may be used in addition to or instead of the depicted hardware. Moreover, the bias monitoring computing system 14 can take any of several different forms of data processing systems, including but not limited to client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants, and the like. In essence, the data processing system 700 can be any known or later developed data processing system without structural limitations.

[0025] Referring back to FIG. 1, each of one or more client devices 12(1) - 12(n) can include a processor, a memory, a user input device (such as a keyboard, mouse, etc.), or an interactive display screen (such as a display device as just one example), or a combination thereof, and a communication interface, which are connected together by a bus or other link, although each can have other types or numbers or both of other systems, devices, components, or other elements, or a combination thereof. In this example, the bias monitoring computing system 14 interacts with one or more client devices 12(1) - 12(n) via a communication network 30 to receive a request to access an application running on one or more application servers 17(1) - 17(n), although the bias monitoring computing system 14 can receive other types or requests.

[0026] Each of one or more training data servers 16(1) to 16(n) can, for example, store training data and provide it to the bias monitoring computing system 14 via one or more of the communication networks 30, although other types or numbers or both of storage media can be used in other configurations. In this particular example, each of one or more training data servers 16(1) to 16(n) can represent a system that includes multiple network server devices within a data storage pool that can include various combinations and types of storage hardware or software or both and can include internal or external networks. Various network processing applications, such as CIFS applications, NFS applications, HTTP web network server device applications, or FTP applications, or combinations thereof, may operate on the multiple data servers 16(1) to 16(n) and can transmit data in response to requests from the bias monitoring computing system 14. Each of one or more training data servers 16(1) to 16(n) can include a processor, a memory, and a communication interface connected together by a bus or other link, although each can have other types or numbers or both of other systems, devices, components, or other elements, or combinations thereof.

[0027] Each of one or more application servers 17(1) to 17(n) can store an application to be executed, for example, via one or more of the communication networks 30, via the bias monitoring computing system 14, and provide access to the application to be executed to one or more client devices 12(1) to 12(n), but other types or numbers or both of storage media can be used in other configurations. In this specific example, each of the plurality of data servers 17(1) to 17(n) can include various combinations and types of storage hardware or software or both, can include internal or external networks, and can represent a system including a plurality of network server devices in a data storage pool. Various network processing applications such as CIFS applications, NFS applications, HTTP web network server device applications, or FTP applications, or combinations thereof, may operate on the plurality of data servers 17(1) to 17(n) and can transmit data in response to requests from the bias monitoring computing system 14. Each of the plurality of data servers 17(1) to 17(n) can include a processor, a memory, and a communication interface connected together by a bus or other link, but each can have other types or numbers or both of other systems, devices, components, or other elements, or combinations thereof.

[0028] A non-transitory computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A non-transitory computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy (R) disk, mechanically encoded devices such as punch cards or raised structures in grooves that record instructions, and any suitable combination of the foregoing. A non-transitory computer-readable storage medium should not be construed as being, as used herein, essentially a transient signal such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through a wire.

[0029] The non-transitory computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices or to an external computer or external storage device via a communication network 30, such as, for example, the Internet, a local area network (LAN), a wide area network (WAN), or a wireless network, or a combination thereof. The communication network 30 can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within each respective computing / processing device.

[0030] The computer-readable program instructions for carrying out the operations of the present invention may be source code or object code written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Java(R), Smalltalk(R), C++, or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of communication network 30 including a LAN or WAN, or the connection may be made to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit device including a programmable logic circuitry, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to customize the electronic circuit device for implementing aspects of the present invention.

[0031] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0032] These computer-readable program instructions may be provided to the processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart or block diagram or both. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes a product having instructions for implementing the aspects of the functions / acts specified in one or more blocks of the flowchart or block diagram or both, and the computer-readable program instructions may direct a computer, programmable data processing apparatus, or other device, or combinations thereof, to function in a particular manner.

[0033] The computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to produce a computer-executed process, such that the instructions which execute on the computer, other programmable apparatus, or other device perform a series of operational steps for implementing the functions / acts specified in one or more blocks of the flowchart or block diagram or both.

[0034] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, segment, or portion of one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may be performed out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams or flowcharts, or combinations of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or act, or by a combination of dedicated hardware and computer instructions.

[0035] This specification and the claims may use the terms "a", "at least one of", and "one or more of" with respect to particular features and elements of the illustrative embodiments. It should be understood that these terms and phrases are intended to state that at least one of the particular features or elements that exist in a particular illustrative embodiment may be present, but more than one may also be present. That is, these terms / phrases are not intended to limit this specification or the claims to a single feature / element that exists, or to require that a plurality of such features / elements must exist. On the contrary, these terms / phrases merely require at least a single feature / element, and a plurality of such features / elements may be within the scope of this specification and the claims.

[0036] Furthermore, it should be understood that the following description further illustrates implementations of examples of the illustrative embodiments using various examples of the various elements of the illustrative embodiments and aids in the understanding of the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and do not cover all the various possibilities for implementing the mechanisms of the illustrative embodiments. Considering this description, it will be apparent to those skilled in the art that there are many other alternative implementations for these various elements in addition to or instead of the examples provided herein without departing from the spirit and scope of the present invention.

[0037] The systems and processes in the figures are not exclusive. Other systems, processes, and menus may be derived in accordance with the principles of the embodiments described herein to achieve the same purpose. It should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the current design may be made by those skilled in the art without departing from the scope of the embodiments. As described herein, various systems, subsystems, agents, managers, and processes may be implemented using hardware components, software components, or combinations thereof, or combinations thereof. The claim elements herein shall not be construed under the provisions of 35 U.S.C 112(f) unless the element is expressly recited using the phrase "means for".

[0038] An exemplary method for detection and monitoring will now be shown with reference to FIGS. 3-6C. Referring in detail to FIG. 3, exemplary method 300 begins at step 305, where bias monitoring computing system 14 indexes the training data to enable instant word search. In this example, indexing the training data can be done by applying a full text search index that enables the calculation of the correlation between the values of the application features and the target variable. By indexing the training data, the disclosed technique can search for feature values that have a high correlation with the target variable and can also be adapted to additional data. In this example, bias monitoring computing system 14 can obtain training data from one of one or more training data servers 16(1)-16(n), although the training data can be obtained from other storage locations. An example of step 305 will now be further shown with reference to FIGS. 4A and 4B. As shown in FIG. 4A, bias monitoring computing system 14 inputs a host name or Internet Protocol (IP) address, a Secure Sockets Layer (SSL) port, and a database existing in one of one or more training data servers 16(1)-16(n) to obtain training data. Next, as shown in FIG. 4B, bias monitoring device 14 selects specific training data from the training table by including schema and table names, although other types of information can be included. In this example, the artificial intelligence model is a supervised learning model, and the model type can be binary classification, multi-class classification, or regression. Further, in this example, the input training data can be structured data.

[0039] Next, in step 310, the bias monitoring computing system 14 determines the correlation between the value of each feature and the target variable using the indexed data and the trained artificial intelligence model shown above in step 305. As an example, the target variable relates to the predicted value by machine learning.

[0040] Next, in step 315, the bias monitoring computing system 14 calculates the difference F in correlation to determine whether each value of the feature is favorable or unfavorable. In this example, favorable relates to the preferred or desirable result of the determination by the machine learning model, and unfavorable relates to the non-preferred or undesirable result of the determination by the machine learning model. As an example, in a binary classification model of loan application approval / rejection, approval corresponds to favorable and rejection corresponds to unfavorable. In this example, the bias monitoring computing system 14 calculates and correlates the first value, the second value, the absolute value, and the total using the formula described below.

[0041]

Equation

[0042] An example of the use of the above formula will now be shown. The favorable and unfavorable values in this example are "no risk" or "risk". As an example, if there are values M and F for the value associated with gender, and there are a total of 50 records, of which 20 records have gender as M and the favorable value is "no risk", 15 records have gender as "F" and the favorable value is "no risk", 10 records have gender M and an unfavorable value as "risk", and 5 records have gender as F and an unfavorable value as risk, calculating with "no risk" as the target variable and gender M as the feature value in the gender data should give a value of 0.95((20) / (20 + 10)) / ((20 + 15) / (50)) = 0.95.

[0043] In step 320, the bias monitoring computing system 14 determines whether the calculation of the correlation has been performed for all features. If the bias monitoring computing system 14 determines that the calculation of the correlation has not been performed for all features, a no branch to step 315 is taken. However, when the bias monitoring computing system 14 determines that the calculation of the correlation has been performed for all features, a yes branch to step 325 is taken.

[0044] In step 325, the bias monitoring computing system 14 presents the trend for each feature and determines whether there is a correlation value, that is, F(f), calculated using the feedback data. If the bias monitoring computing system 14 determines that F(f) exists, a yes branch to step 330 is taken. In this example, the value of F for each feature shown in step 315 is obtained by the following formula using r(feature, favorable / unfavorable) for each value in the feature: F = sum(|r(feature, favorable) - r(feature, unfavorable)|). Therefore, F is determined for each feature, and the trend for each feature is useful for selecting the features to be monitored.

[0045] In step 330, the bias monitoring computing system 14 proposes features that satisfy F(t)*a ≦ F(f) as candidates for the features to be monitored. In this example, F(t) means F calculated from the training data, F(f) means F calculated from the feedback data, the formula is used for each feature, and here a can take any value. The example of step 330 is shown in FIG. 5, and the exemplary flow proceeds to step 340 further shown below. As shown in FIG. 5, the bias monitoring computing system 14 proposes gender and age as features that satisfy the formula shown above.

[0046] However, if in step 325 the bias monitoring computing system 14 determines that F(f) does not exist, a no branch to step 335 is taken. In step 335, the bias monitoring computing system 14 refers to the "r" of the selected feature and, as shown in FIGS. 6A to 6C, proposes values of the selected feature having a high correlation with convenience as majority candidates and proposes values of the selected feature having a high correlation with inconvenience as minority candidates. As an example, the majority relates to a group of feature values that are more likely to contribute to convenience, and the minority relates to a group of feature values that are more likely to contribute to inconvenience. Taking gender as an example of a feature in a binary classification model for loan application approval / rejection, men belong to the majority and women belong to the minority. Alternatively, taking annual income as an example of a feature, people with an annual income of 10 million yen or more in Japanese yen belong to the majority, and people with an annual income of 3 million yen or less in Japanese yen belong to the minority.

[0047] As an example, when the favorable and unfavorable values are "No Risk" and "Risk", "M" and "F" are values associated with gender, the total number of records is 50, among which there are 20 records with "M" as the "gender" column and "No Risk" as the target variable, 15 records with "F" as the "gender" column and "No Risk" as the target variable, 10 records with "M" as the "gender" column and "Risk" as the target variable, and 5 records with "F" as the "gender" column and "Risk" as the target variable. If gender is selected as the monitored feature, the values are calculated as r as follows. r("M", "No Risk") = ((20) / (20 + 10)) / ((20 + 15) / (50)) = 0.95, r("F", "No Risk") = ((15) / (15 + 5)) / ((20 + 15) / (50)) = 1.07, r("M", "Risk") = ((10) / (20 + 10)) / ((10 + 5) / (50)) = 1.11, and r("F", "Risk") = ((5) / (15 + 5)) / ((10 + 5) / (50)) = 0.83. Thus, in this illustrative example, the value of the selected feature that has a high correlation with the favorable ("No Risk") is "F", and thus "F" is proposed as the majority candidate. On the other hand, the value of the selected feature that has a high correlation with the unfavorable ("Risk") is "M", and thus "M" is proposed as the minority candidate.

[0048] Next, in step 340, the bias monitoring computing system 14 provides feedback data, indexes the data, and the exemplary flow proceeds to step 310. In this example, the feedback data is new data, which is in the same format as the training data and should be additional correct answer data predicted by the model in response to the actual input to the model. Further, the feedback data can be provided any number of times.

[0049] By using the techniques shown above, the disclosed techniques can accurately identify features with potential biases that might not otherwise be identified by humans such as data scientists. Further, indexing the training data enables easy addition of new data and supports efficient computations. Even when an error in the setting of the target variable or the like is notified and the setting is changed, preparing an index separate from the actually used model enables dynamic calculation of correlations, and thus the index is useful. Further, the disclosed techniques can also be applied to new data, and thus can handle changes in features according to the trend of the times.

[0050] As an example, when applying the disclosed techniques to training data, age is not selected as a feature to be monitored in the application, and when applying the disclosed techniques to new data called feedback data, if F(t)*a≦F(f) in step 330 is satisfied by the age feature, age is newly recommended as a candidate for the feature to be monitored. Thus, by applying the disclosed techniques each time new data is added, candidates for the features to be monitored are proposed from the latest calculation results, and changes in the trends of each feature are also detected. This technique using new data enables finding features that should not be identified in the training data but have potential biases.

[0051] The present invention has been described with reference to exemplary embodiments, but is not limited to the exemplary embodiments. Those skilled in the art will understand that very many changes and modifications will be made to the preferred embodiments of the present invention, and that such changes and modifications will be made without departing from the true spirit of the present invention. Accordingly, the appended claims are intended to be construed so as to cover all such equivalent variations that fall within the true spirit and scope of the present invention.

Claims

1. A method for monitoring and detecting bias in an application, comprising: indexing training data by a bias monitoring computing system; obtaining, by the bias monitoring computing system, a plurality of correlation values between one or more features and a target variable in the indexed training data; for each of the one or more features, calculating, by the bias monitoring computing system, a first correlation value of the plurality of correlation values between the one of the one or more features and a favorable result of a judgment by a trained artificial intelligence model; calculating, by the bias monitoring computing system, a second correlation value of the plurality of correlation values between the one of the one or more features and an unfavorable result of a judgment by the trained artificial intelligence model; calculating, by the bias monitoring computing system, an absolute value of a difference between the calculated first value and the calculated second value; calculating, by the bias monitoring computing system, a total of the calculated absolute values of the plurality of correlation values for the one of the one or more features; identifying and monitoring bias in the one or more features by the bias monitoring computing system using the calculated first value, the second value, the absolute value, and the total; A method comprising the above steps.

2. The method according to claim 1, further comprising indexing feedback data received from a client device by the bias monitoring computing system.

3. The method according to claim 1, further comprising obtaining training data from a training data server by the bias monitoring computing system.

4. The method according to claim 2, further comprising calculating, by the bias monitoring computing system, a second total of the calculated absolute values of the plurality of correlation values for the one of the one or more features for the feedback data.

5. The method according to claim 4, further comprising presenting a correlation change value determined by a difference between a calculated total of the calculated absolute values associated with the training data and a calculated total of the calculated absolute values associated with the feedback data by the bias monitoring computing system.

6. The method according to claim 5, wherein the correlation change value represents a candidate for a feature to be monitored.

7. A non-transitory computer-readable medium that, when executed by at least one machine, indexes training data, obtains a plurality of correlation values between one or more features and a target variable in the indexed training data, for each of the one or more features, calculates a first value of correlation among the plurality of correlation values between the one of the one or more features and a favorable result of a judgment by a trained artificial intelligence model, calculates a second value of correlation among the plurality of correlation values between the one of the one or more features and an unfavorable result of a judgment by the trained artificial intelligence model, calculates an absolute value of a difference between the calculated first value and the calculated second value, calculates a total of the calculated absolute values of the plurality of correlation values of the one of the one or more features, and uses the calculated first value, the second value, the absolute value, and the total to identify and monitor bias in the one or more features A non-transitory computer-readable medium storing instructions comprising machine-executable code that causes the machine to perform the above.

8. The medium according to claim 7, further comprising indexing feedback data received from a client device.

9. The medium according to claim 7, further comprising obtaining training data from a training data server.

10. The medium according to claim 8, further comprising calculating a second total of the calculated absolute values of the plurality of correlation values of the one of the one or more features for the feedback data.

11. The medium according to claim 10, further comprising presenting a correlation change value determined by a difference between the calculated total of the calculated absolute values associated with the training data and the calculated total of the calculated absolute values associated with the feedback data.

12. The medium according to claim 11, wherein the correlation change value represents a candidate for a feature to be monitored.

13. A bias monitoring computing system, comprising: a memory having programmed instructions stored therein; and one or more processors configured to execute the programmed instructions stored in the memory to index training data, obtain a plurality of correlation values between one or more features and a target variable in the indexed training data, and for each of the one or more features, calculate a first value of a correlation among the plurality of correlation values between the one of the one or more features and a favorable result of a judgment by a trained artificial intelligence model, calculate a second value of a correlation among the plurality of correlation values between the one of the one or more features and an unfavorable result of a judgment by the trained artificial intelligence model, calculate an absolute value of a difference between the calculated first value and the calculated second value, calculate a total of the calculated absolute values of the plurality of correlation values of the one of the one or more features, and identify and monitor bias in the one or more features using the calculated first value, the second value, the absolute value, and the total. A bias monitoring computing system comprising the one or more processors configured to perform the above.

14. The system according to claim 13, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to index feedback data received from a client device.

15. The system according to claim 13, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to obtain training data from a training data server.

16. The system of claim 14, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to calculate a second sum of the calculated absolute values of the one of the one or more features for the feedback data. **Claim 17** The system of claim 16, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to present a correlation change value determined by a difference between the calculated sum of the calculated absolute values associated with the training data and the calculated sum of the calculated absolute values associated with the feedback data. **Claim 18** The system of claim 17, wherein the correlation change value represents a candidate feature to be monitored. **Claim 19** The system of claim 14, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to provide the one or more identified and monitored features with the bias to the client device. **Claim 20** The system of claim 19, wherein the one or more processors are further configured to execute the programmed instructions stored in the memory to provide an advantage or disadvantage determination of the provided one or more features with the bias. **Claim 21** A method for monitoring and detecting bias in an application, comprising: identifying features from one or more features of a software application by a bias monitoring computing system, indexing training data, and identifying the features, including obtaining a plurality of correlation values between one or more features and a target variable in the indexed training data; determining by the bias monitoring computing system whether there is a bias in the identified features from the one or more features of the software application, for each of the one or more features Calculating a first value of correlation among the plurality of correlation values between the one of the one or more features and a favorable result of a judgment by the trained artificial intelligence model; Calculating a second value of correlation among the plurality of correlation values between the one of the one or more features and an unfavorable result of a judgment by the trained artificial intelligence model; Calculating an absolute value of a difference between the calculated first value and the calculated second value; Calculating a total of the calculated absolute values of the plurality of correlation values of the one of the one or more features; Determining whether there is such bias, including identifying and monitoring bias in the one or more features using the calculated first value, second value, absolute value, and total; Determining a favorable or unfavorable determination by the bias monitoring computing system when there is the bias determined in the identified feature; Providing, by the bias monitoring computing system, the identified feature with the determined bias and the determined favorable or unfavorable determination A method comprising.

Images