Examining generative artificial intelligence for bias

Abstract

To provide a system and method for examining a generative artificial intelligence for bias.SOLUTION: A method may include the steps of: obtaining a topic 102 and an artificial intelligence (AI) role 104 relating to the topic; generating, by a primary generative AI model 110, a question prompt 112 based on the topic and AI role; generating, by the primary generative AI model, one or more statements as a statement set 114 corresponding to the question prompt; masking key terms included in the statements; determining, by a secondary generative AI model 150, unmasked statements 155 corresponding to the masked statements 145; and evaluating 165 performance of the secondary generative AI model by comparing the unmasked statements to corresponding statements of the statement set.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for inspecting generative artificial intelligence for bias.

Background Art

[0002] Artificial intelligence (AI) may be used in various research fields to solve problems in those research fields. After being configured and trained to understand such analysis tasks, AI can perform analysis tasks that have conventionally been performed by humans. For example, AI can make content recommendations based on the preferences or input information of one or more users, make comprehensive and non-binary decisions according to a specific set of rules, synthesize or create artworks, or perform other analytical or creative tasks. An AI model can be trained to perform analytical or creative tasks, for example, by providing a knowledge base. From that knowledge base, the AI model can statistically analyze the features of the knowledge base to derive a statistically valid response to new inputs (i.e., a machine learning model). As an additional or alternative example, an AI model can be trained by configuring the AI model to analyze a knowledge base according to a specific logical or structural process to reach a specific conclusion for new inputs (such as a natural language processing model).

[0003] The subject matter claimed in the present disclosure is not limited to embodiments that solve any disadvantages or that operate only in the environments as described above. Rather, this background art is provided only to explain a technical area in which some embodiments described in the present disclosure may be implemented.

Summary of the Invention

[0004] According to one aspect of an embodiment, the method can include obtaining a topic and a role of artificial intelligence (AI) related to the topic, where the topic is related to a research field and the role of the AI represents an occupational role in the research field. The method can include generating a question prompt based on the topic and the role of the AI by a first generation AI model. The method can include generating, by the first generation AI model, one or more statements as a set of statements corresponding to the question prompt, and masking key terms included in the statements where each statement includes at least one respective key term. The method can include determining, by a second generation AI model, an unmasked statement corresponding to the masked statement. The method can include evaluating the performance of the second generation AI model by comparing the unmasked statement with the corresponding statement in the set of statements.

[0005] The objectives and advantages of the embodiments are at least realized and achieved by the elements, features, and combinations particularly pointed out in the claims. It should be understood that both the foregoing general description and the following detailed description are examples and not limitations of the claimed invention.

Brief Description of the Drawings

[0006] Exemplary embodiments are described and explained with further particularity and detail through the following accompanying drawings.

[0007]

Figure 1

[0008]

Figure 2A

[0009]

Figure 2B

[0010]

Figure 3

[0011]

Figure 4

[0012]

Figure 5A

[0013]

Figure 5B

[0014]

Figure 6

[0015]

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0016] A generative artificial intelligence (AI) model can be configured to provide recommendations and answers to a user's inquiry regarding a specific topic. The generative AI model is trained on a knowledge base related to a specific topic and can establish connections between different ideas related to the specific topic. The natural language processing aspect of the generative AI model can facilitate the analysis of questions or requests raised by the user and the association of the analyzed information with the knowledge base of the generative AI model. Based on how the analyzed user input relates to the knowledge base, the generative AI model can return a response likely to satisfy the user's question or request.

[0017] However, the generative AI model may be trained on a knowledge base that contains incorrect information (such as misinformation or discriminatory information) regarding a specific topic. Additionally or alternatively, the knowledge base may contain biased information. In topics where policy decisions and trade-offs between two or more options may be involved, the biased information contained in the knowledge base can distort the analysis provided by the generative AI model and result in a bias in the outcomes. For example, a user's request or question regarding a country's education policy may rely on a knowledge base that undervalues one or more minority groups, and as a result, the conclusion of the output by the generative AI may also undervalue one or more minority groups. Therefore, it is necessary to audit a specific generative AI model and evaluate whether the output of the specific generative AI model is biased or unfair.

[0018] Additionally or alternatively, a particular generative AI model may be improved by evaluating how or whether the output of the particular generative AI model is biased or unfair so that additional information can be introduced into the knowledge base corresponding to the particular generative AI model to correct the identified bias. Evaluating a generative AI model according to the present disclosure can enable more effective retraining of a biased AI model by improving the recognition of how the output presented by the generative AI model is biased or otherwise distorted. Such an evaluation of a particular generative AI model can additionally or alternatively enable the identification of which aspects of the knowledge base used to train the particular generative AI model may contain biased information. A particular generative AI model can be improved by retraining the particular generative AI model using a knowledge base from which the biased information has been excluded or using a new knowledge base that includes information that cancels out the biased information. Additionally or alternatively, the output of a particular generative AI model can be improved by aligning the performance of the particular generative AI model with the performance of one or more generative AI models against which the particular generative AI model is compared in response to a determination that the particular generative AI model outputs biased or discriminatory information.

[0019] Embodiments of the present disclosure are described with reference to the accompanying drawings.

[0020] FIG. 1 is a diagram of an exemplary operating environment 100 that can evaluate the fairness or bias of a generative artificial intelligence (AI) model, according to one or more embodiments of the present disclosure. The environment 100 can include a primary generative AI model 110 configured to output a question prompt 112 and a set of statements 114. The set of statements 114 includes true statements 116 and false statements 118 based on a topic 102 provided to the primary generative AI model 110 and a given AI role 104. A masking module 140 can generate a masked statement 145 based on the set of statements 114. The masked statement 145 can be used by a secondary generative AI model 150 to generate a corresponding unmasked statement 155 that un-masks a corresponding masked word or phrase within the masked statement 145. An evaluation module 160 can analyze words selected to be included in the unmasked statement 155 to determine a secondary generative AI evaluation 165.

[0021] In some embodiments, the primary generation AI model 110, the masking module 140, the secondary generation AI model 150, and / or the evaluation module 160 (collectively referred to herein as the "computing module") can include code and routines configured to enable a computing system to perform one or more operations. Additionally or alternatively, one or more of the computing modules may be implemented using hardware including a processor, a microprocessor (e.g., one that performs or controls the performance of one or more operations), an FPGA (field-programmable gate array), an ASIC (application-specific integrated circuit), a GPU (graphics processing unit), or a TPU (tensor processing unit). In some other examples, the computing module may be implemented using a combination of hardware and software. In the present disclosure, operations described as being performed by the computing module may include operations that the computing module may instruct one or more corresponding systems to perform. The computing module may be configured to perform a series of operations as further described below in connection with the exemplary method 600 described with respect to FIG. 6 with respect to the topic 102, the role of AI, the question prompt 112, the statement set 114, the masked statement 145, the unmasked statement 155, and / or the secondary generation AI evaluation 165.

[0022] Topic 102 and the role of AI 104 may be provided to the primary generation AI model 110 so that the primary generation AI model 110 can return a question prompt 112 and a statement set 114. In some embodiments, Topic 102 can include one or more fields of technology or research where the fairness of a decision maker is considered important for the results in the field of technology or research. For example, Topic 102 can include gender equality, high-quality education, climate change mitigation, housing development, or any other topic where fair results are desired.

[0023] The role of AI 104 can include a set of one or more roles that the primary generation AI model 110 can play when analyzing Topic 102, and the analysis of Topic 102 by the primary generation AI model 110 can result in outcomes that may or may not ultimately be considered fair. For example, the role of AI 104 corresponding to gender equality can include being a discrimination advisor, and the role of AI 104 corresponding to climate change mitigation can include being a climate fairness advocate. As an additional or alternative example, the role of AI 104 corresponding to high-quality education can include being a college admissions evaluator. As an additional or alternative example, the role of AI 104 can include functioning as a security advisor regarding source code review. In these and other embodiments, Topic 102 and the role of AI 104 may be provided by a human user. As an additional or alternative, Topic 102 may be provided by a human user, and one or more roles of AI 104 may be proposed by an AI model such as the primary generation AI model 110.

[0024] Based on Topic 102 and the role of AI 104, the question prompt 112 may be output by the primary generation AI model 110. In some embodiments, the question prompt 112 may be a question that may result in responses having different degrees of accuracy, usefulness, or fairness if the response is implemented in a real-world environment. For example, a particular question prompt 112 may be, "Should paternity leave be provided as a standard benefit to all working fathers?" As an additional or alternative example, a particular question prompt 112 may be, "How can we address the issues of climate refugees and asylum seekers?"

[0025] FIG. 2A shows a flowchart of an exemplary process 200a for generating a generative AI input 210 for use in evaluating the fairness and bias of a generative AI model, according to one or more embodiments of the present disclosure. Process 200a may include creating a question prompt template 208 based on an AI role 202 that may be the same as or similar to the role of AI 104, a topic 204 that may be the same as or similar to topic 102, and a configuration 206 that can specify a customization process for the question prompt template 208 based on a predetermined set of possible topics.

[0026] In some embodiments, the question prompt template 208 may provide a framework for user input that can be provided to an AI model, such as a primary generation AI model 220, to return a corresponding output. For example, the question prompt template 208 can be described as follows: "If you <airole>Assume that it is. When you are given a set of statements, <topic>has the responsibility to evaluate the ethics. In the form of a list of Python strings <number>Generate only a few questions. Do not add explanations. Top <number>What are the questions?」 In these and other embodiments, the generative AI input 210 may include filling in the blank terms included in the question prompt template 208 using the role 202 and topic 204 of the AI. Additionally or alternatively, the generative AI input 210 can be specified based on further user input, such as a user input that specifies the number of questions to be returned by the primary generative AI model 220. Based on the generative AI input 210, the primary generative AI model 220 can return a number of subtopic questions 225 in response to the generative AI input 210. The subtopic questions 225 may be the same as or similar to the question prompt 112 described with respect to FIG. 1.

[0027] Returning to FIG. 1, the question prompt 112 may be a question that can obtain true statements 116 and false statements 118 collectively referred to as the statement set 114. FIG. 2B shows a flowchart of an exemplary process 200b for generating true and false statements for use in evaluating the fairness and bias of a generative AI model according to one or more embodiments of the present disclosure. The process 200b may include generating a true sentence prompt template 232 and a false sentence prompt template 234 based on the configuration 206. Applying the subtopic questions 225 to the true sentence prompt template 232 and the false sentence prompt template 234 can generate true statements 236 and false statements 238 that are the same as or similar to the true statement 116 and false statement 118 of FIG. 1, respectively.

[0028] Returning to the description of the environment 100 of FIG. 1, the statement set 114 can be generated based on the question prompt 112, as described in connection with the process 200b of FIG. 2B. In some embodiments, one or more true statements 116 and one or more false statements 118 can be provided via a manual statement input 120 that can be generated from a human user. Providing a mixture of human-generated statements and statements generated by the primary generation AI model 110 can achieve a more detailed evaluation of the performance of the secondary generation AI model 150, as will be described in more detail below in connection with the evaluation module 160 and the secondary generation AI evaluation 165.

[0029] In these and other embodiments, the statement evaluator 130 can review the statement set 114 and audit the generated true statements 116 and the generated false statements 118. The statement evaluator 130 can reject or update any statement included in the statement set 114 that is incorrect, biased, or otherwise inconsistent with the question prompt 112, the role of the AI 104, or the topic 102.

[0030] FIG. 3 shows a flowchart of an exemplary process 300 for evaluating the validity of true and false statements by a statement evaluator 130, etc. of FIG. 1, according to one or more embodiments of the present disclosure. As shown in process 300, questions 310 can be used to generate true statement 312 and false statement 314 that are identical or similar to true statement 116 and false statement 118, respectively. An evaluator 320 that can perform tasks identical or similar to those performed by statement evaluator 130 can determine whether true statement 312 and false statement 314 are correct. In response to determining that a particular true statement 312 or a particular false statement 314 is correct, the particular true statement 312 or false statement 314 can be sent to a masking module 330 that is identical or similar to the masking module 140 described in connection with FIG. 1. In response to determining that a particular true statement 312 or a particular false statement 314 is incorrect, the particular true statement 312 or false statement 314 can be ignored by a masking process performed by masking module 330.

[0031] Returning to the description of the environment 100 of FIG. 1, to output the masked statement 145, the statement set 114 can be sent to the masking module 140. The masking module 140 can be configured to identify terms (e.g., words) within a particular statement included in the statement set 114 and determine whether the identified term is a key term that specifies the meaning of the particular statement. In some embodiments, the determination of whether a particular term is a key term can be realized by referring to a dictionary of stop words. Stop words represent common words and phrases that may exist in various different statements. For example, words such as "a", "an", "and", "are", "as", "at", "be", "by", "for", "from", "has", "he", "in", "is", "it", "its", "of", "on", "that", "the", "to", "was", "were", "will", "with", etc. can be regarded as stop words, among other things. Whether a particular word is regarded as a stop word can depend on the collection frequency of the particular word, which indicates the total number of times the particular word appears in a collection of words. Given the statement set 114, the 5, 10, 20, 100, 500, or any other arbitrary number of words that appear most frequently within the statement set 114 can be identified as stop words. In these and other embodiments, the identification of stop words can be adjusted according to a particular language by configuring the masking module 140 to identify potential stop words in different languages. Additionally or alternatively, stop words identified in a first language can be translated and classified as stop words in a second language.

[0032] For a particular statement, the masking module 140 can identify terms associated with the particular statement that do not match any stop words as key terms. The masking module 140 can execute a masking process for each of the key terms of the particular statement to generate a masked statement 145. Thus, for each true statement 116 and each false statement 118, a plurality of masked statements 145 can be generated, and the number of masked statements 145 depends on the number of key terms identified in the statement set 114. For example, a particular statement included in the statement set 114 as the true statement 116 can be "According to research, it has been shown that black women have more difficulty accessing and advancing in their careers." The masking module 140 can identify "black women", "difficulty", "hardship", "access", "advancement", or "career" as keywords within the particular statement. Next, a particular masked statement 145 can be "According to research, <mask>It can be stated that "it has been shown that it is more difficult to access one's occupation and advance."

[0033] The secondary generation AI model 150 can obtain the masked statement 145 and predictively output the unmasked statement 155. In some embodiments, the secondary generation AI model 150 may be a different AI model from the primary generation AI model 110 used to generate the question prompt 112 and the statement set 114. This is because the unmasked statement 155 is likely to be the same as or similar to the statements included in the statement set 114 before masking if it is generated by the primary generation AI model 110. In other words, when evaluating the primary generation AI model 110 with respect to the statements generated by the primary generation AI model 110 itself, it may not be effective to evaluate the bias and potential unfairness of the primary generation AI model 110. Instead, the secondary generation AI model 150 is used to generate the unmasked statement 155 so that the bias of the secondary generation AI model 150 can be evaluated. In these and other embodiments, multiple secondary generation AI models 150 are used to generate each set of unmasked statements 155, and each set of unmasked statements 155 is compared with the statement set 114 to evaluate the bias of each of the multiple secondary generation AI models 150. Therefore, different secondary generation AI models 150 can be comparatively evaluated by the evaluation module 160.

[0034] Unmasking a masked statement 145 as an unmasked statement 155 can include requiring a particular second-generation generative AI model 150 to take as input a particular masked statement 145 and output one or more terms that can be used to fill in the masked terms included in the particular masked statement 145. In some embodiments, the particular masked statement 145 can be provided as input to the second-generation generative AI model 150 along with a topic 102 and the role of the AI 104 to provide context to the second-generation generative AI model 150. The second-generation generative AI model 150 may be prompted to output several words or phrases that can replace the masked terms of the particular masked statement 145. Additionally or alternatively, the second-generation generative AI model 150 may be prompted to output the probability of each of several words or phrases that can be used to replace the masked terms of the particular masked statement 145. Given a particular masked statement 145, for example, the second-generation generative AI model 150 can return "black woman", "men", and "he". As an additional or alternative example, the second-generation generative AI model 150 can return "black woman: 0.98", "men: 0.77", and "he: 0.66" to indicate the relative probability of the second-generation generative AI model 150 outputting the aforementioned responses for the particular masked statement 145.

[0035] In this example and other examples, different second-generation generative AI models 150 can return, for example, "Male: 0.96", "Black female: 0.92", and "He: 0.81". Since a specific statement originally included the term "Black female", the second-generation generative AI model 150 (the first AI model) that returned "Black female: 0.98" can be considered fairer and less biased than the second-generation generative AI model 150 (the second AI model) that returned "Black female: 0.92" with respect to a specific masked statement 145. This is because the first AI model is more likely to return the masked term than the second AI model.

[0036] The unmasking process performed by the second-generation generative AI model 150 can be executed for each masked term included in each of the masked statements 145. Such a process can be executed iteratively and / or in parallel for different masked statements. FIG. 4 shows a flowchart of an exemplary process 400 that masks key terms included in true and false statements and iteratively unmasks the masked key terms according to one or more embodiments of the present disclosure. A masking module 410 identical or similar to the masking module 140 of FIG. 1 can mask the key terms included in a specific statement in block 411. A specific statement containing a masked term can be unmasked by a second-generation generative AI model 420 that may be identical or similar to the second-generation generative AI model 150 of FIG. 1 in block 412.

[0037] In block 413, the secondary generation AI model 420 can quantify how accurately it un-masks key terms as a score and corresponding rank, and the masking module 410 can determine whether it can execute additional masking procedures in block 414 based on whether a particular statement further contains key terms. In response to the determination that a particular statement contains at least one or more key terms, process 400 returns to block 411, and the next key term is masked by the masking module 410. In response to the determination that a particular statement no longer contains key terms, process 400 proceeds to block 415, where it can determine whether there are additional statements to be masked. In response to the determination that there is at least one additional statement within a particular set of statements, process 400 returns to blocks 411 - 414 and the secondary generation AI model 420 for the remaining statements and can perform the operations described above for the remaining statements. In response to the determination that there are no further statements within a particular set of statements, process 400 proceeds to block 416, where the masking operation ends. Although shown as an iterative process, method 400 can be modified to execute one or more operations in parallel so that method 400 can be executed in parallel at once and / or in a hybrid iterative and parallel manner for all masked statements corresponding to the same statement.

[0038] Returning to the description of FIG. 1, the unmasked statement 155 can be obtained by the evaluation module 160 and can return a secondary generation AI evaluation 165. The evaluation module 160 can be configured to quantitatively score the unmasking performance of the secondary generation AI model 150. In a situation where multiple secondary generation AI models 150 perform an unmasking operation, the evaluation module 160 can rank the secondary generation AI models 150 according to the calculated scores and can be configured to determine the most fair and / or least biased AI model among the tested secondary generation AI models 150.

[0039] In some embodiments, the evaluation module 160 can calculate the evaluation score of a particular tested secondary generation AI model 150 by first determining the likelihood that a particular secondary generation AI model 150 will return the correct unmasked term for a particular masked statement and the likelihood that a particular secondary generation AI model 150 will return an incorrect unmasked term. The evaluation module 160 can repeatedly calculate the scores of other masked terms included in a particular masked statement. Additionally or alternatively, the evaluation module 160 can repeatedly calculate for other masked statements and, for each question prompt associated with a particular topic, repeatedly calculate the score. By calculating scores across multiple masked terms, masked statements, and question prompts, the fairness and bias of a particular secondary generation AI model 150 can be evaluated across multiple different situations and topic areas.

[0040] In these and other embodiments, the calculation of the score of a particular secondary generation AI model 150 for a particular masked term of a particular masked statement can be represented by the following equation:

Equation

[0041] Additionally or alternatively, the calculation of the score for a particular secondary generation AI model 150 can be represented by an expression that includes the ranking of unmasked terms rather than the probability of predicting the correct masked term: [Number] Here, the score A R (.) for a particular masked statement is calculated based on the rank of the correct masked term predicted by the secondary generation AI model 150, assuming that the secondary generation AI model 150 returns the number η of terms that could be unmasked. Similar to the score described in relation to Equation (1), the score in Equation (2) can be averaged over the total number of masked terms.

[0042] Therefore, the overall evaluation of fairness and bias for a particular secondary generation AI model 150 with respect to a set of topics can be represented by the following equation: [Number] Here, the overall evaluation score Eval i for a particular secondary generation AI model 150M Mi is calculated as the sum of the individual scores A(S n k,l ,M i ) for L masked terms, K masked statements, and N question prompts for the particular secondary generation AI model 150.

[0043] Calculating the overall assessment of fairness and bias of a particular second-generation generative AI model 150 according to Equation (3) or any other evaluation metric realizes identifying how the particular second-generation generative AI model 150 outputs biased results. For example, if the overall evaluation score is low, it indicates that the particular second-generation generative AI model 150 has low performance in generating unbiased or fair results for a particular topic, and if the overall evaluation score is high, it indicates that the particular second-generation generative AI model 150 provides unbiased results. In response to receiving a low overall evaluation score for a particular topic, an updated knowledge base can be provided to the second-generation generative AI model 150 and the second-generation generative AI model 150 can be retrained with respect to the particular topic. Additionally or alternatively, issues related to the knowledge base used to train the second-generation generative AI model 150 may be identified in response to a determination that the second-generation generative AI model 150 is outputting skewed results with respect to a particular topic.

[0044] In some embodiments, the overall evaluation score of the first second-generation generative AI model 150 can be compared with the overall evaluation score of the second second-generation generative AI model 150 to determine which generative AI model provides more skewed results with respect to a particular topic. The knowledge base and training process associated with the generative AI model that provides more skewed results may be modified and improved based on the knowledge base and training process associated with the generative AI model that provides less skewed results.

[0045] Changes, additions, or omissions may be made to environment 100 without departing from the scope of the present disclosure. For example, the designation of different elements as described is meant to help explain the concepts described herein and is not limiting. For example, in some embodiments, topic 102, the role of AI 104, question prompt 112, statement set 114, masked statement 145, unmasked statement 155, and secondary generation AI evaluation 165 are depicted in a particular way to help explain the concepts described herein, but such depictions are not limiting. Further, environment 100 may include any number of other elements or may be implemented within a system or environment other than those described.

[0046] In some embodiments, the secondary generation AI evaluation 165 of FIG. 1 may be compiled and visually presented for a user to review and analyze. For example, FIG. 5A shows a first exemplary dashboard 500 configured to present an evaluation of a generative AI model for user analysis in accordance with one or more embodiments of the present disclosure. As shown, dashboard 500 includes a role filter option 502, a true / false statement filter option 504, and a metric presentation option 506 that enables a user to filter and rearrange the data presented on dashboard 500. Dashboard 500 may include a data presentation section 510 that presents the role 512 of the tested AI model in relation to the evaluation score 514 of the corresponding role, as shown in FIG. 5A. Since the average score of the false statements is shown in data presentation section 510, the user can determine that the tested AI model appears to be less likely to make mistakes during the unmasking process when acting as an expert in sustainable energy or an advocate for social justice overall. Conversely, the tested AI model was more likely to make mistakes, on average, when fulfilling the role of a climate scientist.

[0047] FIG. 5B shows a second exemplary dashboard 520 configured to present an evaluation of a generative AI model for user analysis, according to one or more embodiments of the present disclosure. The dashboard 520 can include filtering options such as source model settings 521, target model settings 522, true / false statement filtering 523, filter type 524, and metric presentation options 525. The data presentation section 530 can visually display the performance of a first AI model labeled "bert-large-uncased" compared to a second AI model labeled "xlm-roberta-base" across a range of questions 532, based on the rank 534 in each question 532. Since the true / false statement filtering option 523 is set to display false statements, the user can observe that the second AI model is superior to the first AI model in each question category 532.

[0048] FIG. 6 is a flowchart of an exemplary method 600 for evaluating fairness and bias in a generative AI model, according to one or more embodiments of the present disclosure. The method 600 may be executed by any suitable system, device, or apparatus. For example, the graph creation module 120, secondary modeling module 140, and / or QAP solver 160 of FIG. 1 can perform one or more operations related to the method 600. Although shown as separate blocks, the steps and operations associated with one or more of the blocks of the method 600 may, depending on a particular implementation, be divided into additional blocks, combined into fewer blocks, or removed.

[0049] Method 600 may start at block 602, where the topic and the role of AI related to the topic can be obtained. In some embodiments, the topic may be a technical or research field where the fairness of the decision maker is considered important for the results in the technical or research field. In these and other embodiments, the role of AI can include one or more roles that the AI model generated can play when analyzing the topic for fairness or bias.

[0050] At block 604, the first generated AI model can generate a question prompt based on the obtained topic and the role of AI related to the topic. The question prompt can be a question that a human assuming the role of AI is likely to raise about the obtained topic.

[0051] At block 606, the first generated AI model can generate a set of statements corresponding to the question prompts included in the prompt set. In some embodiments, the set of statements can include true statements and false statements regarding the question prompts. In some embodiments, the set of statements can include statements generated by the first generated AI model in addition to statements manually generated by a human user (e.g., statements provided by an expert user or information provided by an expert user such as published materials provided by an expert user).

[0052] At block 608, the key terms included in the statements of the statement set can be masked to form a set of masked statements. In some embodiments, each statement of the statement set can include respective key terms, and each masked statement included in a particular set can include at least one masked key term. Masking the key terms included in the statements of the statement set can include identifying one or more stop words that represent common words included in the natural language processing of the statement set, and filtering and removing the stop words from each statement. The key terms included in each statement can be identified based on the words remaining in each statement after filtering and removing the stop words. In these and other embodiments, one or more of the identified key terms can be masked.

[0053] At block 610, the second generation AI model can determine a set of unmasked statements. Each unmasked statement within this set is based on the corresponding masked statement. In some embodiments, the determination of the set of unmasked statements can be performed by both the second generation AI model and a third generation AI model that is different from the second generation AI model. In these and other embodiments, the performance of the second generation AI model and the third generation AI model can be evaluated at block 612, where the performance of the second generation AI model and the third generation AI model are compared with respect to fairness and bias to comparatively evaluate the performance of the AI models.

[0054] In block 612, the performance of the second generation AI model can be evaluated based on comparing a set of unmasked statements with each statement included in the set of statements. In some embodiments, based on the performance evaluation of the second generation AI model, the second generation AI model can be retrained using a second training data set that is different from the first training data set initially used to train, retrain, or fine-tune the second generation AI model, or it may not be necessary to do so.

[0055] For example, the performance of the second generation AI model can include calculating an evaluation score that quantitatively indicates how well the second generation AI model determines a set of unmasked statements based on the probability that the second generation AI model returns the total number of correct masked terms and masked key terms included in a particular statement. As an additional or alternative example, evaluating the performance of the second generation AI model can include calculating an evaluation score that quantitatively indicates how appropriately the second generation AI model determines a set of unmasked statements based on the total number of unmasked key terms included in a particular statement, and a ranking of the frequency of correct unmasked key terms returned by the second generation AI model relative to the frequency of incorrect unmasked key terms returned by the second generation AI model. In these and other embodiments, the second generation AI model can be retrained in response to the calculation of an evaluation score indicating that the second generation AI model is providing a biased output.

[0056] Changes, additions, or omissions may be made to method 600 without departing from the scope of the present disclosure. For example, the designation of different elements as described is meant to assist in explaining the concepts described herein and is not limiting. Further, method 600 may include any number of other elements or may be implemented within a system or environment other than those described.

[0057] FIG. 7 is an exemplary computer system 700 according to one or more embodiments of the present disclosure. Computing system 700 may include a processor 710, a memory 720, a data storage device 730, and / or a communication unit 740. All of these may be communicatively coupled. Any or all of the environments 100 of FIG. 1 may be implemented as a computing system that is consistent with computing system 700.

[0058] Typically, processor 710 may include any suitable dedicated or general-purpose computer, computing entity, or processing device that includes various computer hardware or software modules, and may be configured to execute instructions stored on any suitable computer-readable storage medium. For example, processor 710 may include a microprocessor, a microcontroller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), a tensor processing unit (TPU), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data.

[0059] Although a single processor is shown in FIG. 7, it is understood that processor 1010 may include any number of processors that are distributed across any number of networks or physical locations and configured to perform any number of operations described herein, individually or jointly. In some embodiments, processor 710 may interpret and / or execute program instructions and / or process data stored in memory 720, data storage device 730, or both memory 720 and data storage device 730. In some embodiments, processor 710 may fetch program instructions from data storage device 730 and load the program instructions into memory 720.

[0060] After the program instructions are loaded into memory 720, processor 710 may execute the program instructions, such as instructions that cause computing system 700 to perform the operations of method 600 of FIG. 6. For example, computing system 700 may execute the program instructions to obtain topics and AI roles, generate question prompts, generate a set of statements, mask key terms included in each statement of the set of statements, determine a set of unmasked statements, and / or evaluate the performance of the second generation AI model.

[0061] Memory 720 and data storage device 730 may include a computer-readable storage medium or one or more computer-readable storage media having computer-executable instructions or data structures stored thereon. Such computer-readable storage media may be any commercially available media that can be accessed by a general purpose or special purpose computer such as processor 710. For example, memory 720 and / or data storage device 730 may include topic 102, AI role 104, question prompt 112, statement set 114, masked statement 145, unmasked statement 155, and / or second generation AI evaluation 165 of FIG. 1. In some embodiments, computing system 700 may include either or neither of memory 720 and data storage device 730.

[0062] By way of example, and not limitation, such a computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disk storage device, a magnetic disk storage device, or other magnetic storage device, a flash memory element (e.g., a solid state memory device), or any other storage medium that can be used to store the desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, including combinations thereof. The foregoing combinations may also be included within the scope of computer-readable storage media. Computer-executable instructions may include, for example, instructions and data configured to cause a processor 710 to perform a particular operation or a set of operations.

[0063] The communication unit 740 may include any component, device, system, or combination thereof configured to transmit or receive information via a network. In some embodiments, the communication unit 740 may communicate with other locations, devices in the same location, or other components within the same system. For example, the communication unit 740 may include a modem, a network card (wireless or wired), an optical communication device, an infrared communication device, a wireless communication device (e.g., an antenna), and / or a chipset (e.g., a Bluetooth device, an 802.6 device (e.g., a metropolitan area network (MAN)), a WiFi device, a WiMax device, cellular communication equipment, etc.), among others. The communication unit 740 may enable data exchange with a network and / or any other device or system described in the present disclosure. For example, the communication unit 740 may enable the system 700 to communicate with other systems, such as communication devices and / or other networks.

[0064] One of ordinary skill in the art, after reviewing the present disclosure, can understand that changes, additions, or omissions can be made to system 700 without departing from the scope of the present disclosure. For example, system 700 may include more or fewer components than those explicitly illustrated and described.

[0065] The foregoing disclosure is not intended to limit the present invention to the disclosed detailed form or a particular field of use. Accordingly, various alternative embodiments and / or modifications to the present disclosure are considered possible in light of the present disclosure, whether or not explicitly described or shown herein. Thus, it is understood that by describing embodiments of the present disclosure, changes may be made formally and in detail without departing from the scope of the present disclosure. Accordingly, the present disclosure is limited only by the claims.

[0066] In some embodiments, components, modules, engines, and services different from those described herein may be implemented as objects or processes (e.g., separate threads) running on a computing system. Although some of the systems and processes described herein are described as being generally implemented in software (stored in and / or executed by general-purpose hardware), dedicated hardware implementations or combinations of software and dedicated hardware implementations are also possible and contemplated.

[0067] The terms used in this disclosure and in particular in the appended claims (the body of the appended claims) are generally to be construed as "broad terms" (e.g., the term "comprising" should be construed as "including but not limited to").

[0068] Furthermore, if an enumeration of a specific number of introduced claims is intended, such intention shall be explicitly indicated in the claims, and if there is no such enumeration, such intention does not exist. For example, for the sake of assistance in understanding, the following appended claims may include the use of introductory phrases "at least one" and "one or more" to introduce an enumeration of claims. However, the use of such phrases does not mean that when the same claim includes the introductory phrase "one or more" or "at least one" and the indefinite article "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more"), the introduction of an enumeration of claims by the indefinite article "a" or "an" should be considered to limit any particular claim including such introduced enumeration of claims to an embodiment including only one such enumeration. That is, the same applies to the use of the definite article used to introduce an enumeration of claims.

[0069] Furthermore, if an enumeration of a specific number of introduced claims is explicitly recited, one of ordinary skill in the art should understand that such enumeration should be construed to mean at least the recited number (e.g., the recitation of "two enumerations" without other modifiers means at least two enumerations, or two or more enumerations). Further, in examples where a recitation such as "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, typically such a configuration is intended to include A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together, etc.

[0070] Furthermore, any disjunctive word or phrase representing two or more alternative terms should be understood to contemplate the possibility of including one of the terms, any of the terms, or both terms, regardless of whether it is in the description, claims, or drawings. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B" or "A and B".

[0071] All of the examples and conditional language recited in this disclosure are intended for the teaching purpose of assisting the reader in understanding the disclosure and the concepts that the disclosure contributes to the further development of the technology, and should not be construed as being limited to such specifically recited examples and conditions. Although the embodiments of the present disclosure have been described in detail, various changes, alternatives, and selections can be made to them without departing from the spirit and scope of the present disclosure.

[0072] In addition to the above embodiments, the following appendices are further disclosed. (Appendix 1) Obtaining a topic and the role of artificial intelligence (AI) related to the topic, wherein the topic is related to a research field, and the role of the AI represents an occupational role in the research field specified by a human user for the topic and the role of the AI; Generating a question prompt based on the topic and the role of the AI by a first generation AI model; Generating one or more statements as a set of statements corresponding to the question prompt by the first generation AI model; Masking key terms included in the statements of the set of statements to form a set of masked statements, wherein each statement of the set of statements includes each key term, and each masked statement included in a specific set includes at least one masked key term; Determining a set of unmasked statements by a second generation AI model, wherein each unmasked statement is based on the corresponding masked statement; Evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the set of statements; A method comprising. (Appendix 2) Evaluating the performance of the second generation AI model is based on comparing the set of unmasked statements with each statement included in the statement set, and based on indicating that the second generation AI model provides inaccurate results, using a second training dataset different from the first training dataset initially used to train the second generation AI model to retrain or fine-tune the second generation AI model. The method according to appendix 1, further comprising. (Appendix 3) One or more statements among the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt. The method according to appendix 1. (Appendix 4) To form the set of masked statements, the step of masking the key terms included in the statements of the statement set is Identifying one or more stop words representing common words included in the natural language processing of the statement set; Excluding the one or more stop words from each statement in the statement set; Based on the words remaining in each statement after excluding the one or more stop words, identifying the key terms included in each statement; Masking one of the identified key terms; The method according to appendix 1, comprising. (Appendix 5) The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score based on the probability that the second generation AI model returns the correct masked key term to replace a specific masked key term included in a specific statement, and the total number of masked key terms included in the specific statement, according to the method described in Appendix 1. (Appendix 6) The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score, and the evaluation score is based on the total number of masked key terms included in a specific statement, and the ranking of the frequency of the correct unmasked key term used to replace a specific masked term being returned by the second generation AI model to the frequency of the incorrect unmasked key term used to replace the specific masked term being returned by the second generation AI model, according to the method described in Appendix 1. (Appendix 7) The step of determining the set of unmasked statements is performed by the second generation AI model and the third generation AI model, The step of evaluating the performance of the second generation AI model includes the step of visually representing the fairness of the second generation AI model in comparison with the fairness of the third generation AI model, according to the method described in Appendix 1. (Appendix 8) One or more non-transitory computer-readable storage media configured to store instructions, which, in response to being executed, cause a system to perform operations, and the operations are The step of obtaining a topic and the role of artificial intelligence (AI) related to the topic, where the topic is related to a research field, and the role of the AI represents an occupational role in the research field where the topic and the role of the AI are specified by a human user, The step of generating a question prompt based on the topic and the role of the AI by the first generation AI model, generating, by the first generation AI model, one or more statements as a set of statements corresponding to the question prompt; masking, to form a set of masked statements, key terms included in the statements of the set of statements, wherein each statement of the set of statements includes respective key terms, and each masked statement included in a particular set includes at least one masked key term; determining, by a second generation AI model, a set of unmasked statements, wherein each unmasked statement is based on a corresponding masked statement; evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the set of statements; one or more non - transitory computer - readable storage media including the above. (Appendix 9) The operations are based on the evaluation of the performance of the second generation AI model indicating that the second generation AI model provides inaccurate results based on comparing the set of unmasked statements with each statement included in the set of statements, re - training or fine - tuning the second generation AI model using a second training data set different from the first training data set initially used to train the second generation AI model; one or more non - transitory computer - readable storage media according to Appendix 8, further including the above. (Appendix 10) One or more of the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt, one or more non-transitory computer-readable storage media described in Appendix 8. (Appendix 11) To form the set of masked statements, the step of masking key terms included in the statements of the statement set comprises identifying one or more stop words representing common words included in the natural language processing of the statement set; excluding the one or more stop words from each statement of the statement set; identifying the key terms included in each statement based on the words remaining in each statement after excluding the one or more stop words; masking one of the identified key terms; and one or more non-transitory computer-readable storage media described in Appendix 8. (Appendix 12) The step of evaluating the performance of the second generation AI model includes calculating an evaluation score based on the probability that the second generation AI model returns the correct masked key term to replace a specific masked key term included in a specific statement and the total number of masked key terms included in the specific statement, one or more non-transitory computer-readable storage media described in Appendix 8. (Appendix 13) The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score, and the evaluation score is based on the total number of masked key terms included in a specific statement, the frequency of correct unmasked key terms used to replace a specific masked term being returned by the second generation AI model, the frequency of incorrect unmasked key terms used to replace the specific masked term being returned by the second generation AI model, and a ranking of the frequencies, on one or more non-transitory computer-readable storage media described in Appendix 8. (Appendix 14) The step of determining the set of unmasked statements is performed by the second generation AI model and the third generation AI model. The step of evaluating the performance of the second generation AI model includes the step of visually representing the fairness of the second generation AI model in comparison to the fairness of the third generation AI model, on one or more non-transitory computer-readable storage media described in Appendix 8. (Appendix 15) A system comprising: One or more processors; One or more non-transitory computer-readable storage media configured to store instructions; wherein the instructions, when executed, cause the system to perform operations, and the operations include: Obtaining a topic and a role of artificial intelligence (AI) related to the topic, wherein the topic is related to a research field, and the role of the AI represents an occupational role in the research field where the topic and the role of the AI are specified by a human user; Generating a question prompt based on the topic and the role of the AI by a first generation AI model; Generating one or more statements as a set of statements corresponding to the question prompt by the first generation AI model; To form a set of masked statements, a step of masking key terms included in the statements of the statement set, wherein each statement of the statement set includes respective key terms, and each masked statement included in a specific set includes at least one masked key term, the step; A step of determining a set of unmasked statements by a second generation AI model, wherein each unmasked statement is based on a corresponding masked statement; A step of evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the statement set; A system including the above. (Appendix 16) The operation is Based on the evaluation of the performance of the second generation AI model indicating that the second generation AI model provides inaccurate results based on comparing the set of unmasked statements with each statement included in the statement set, using a second training dataset different from the first training dataset initially used to train the second generation AI model, a step of retraining or fine-tuning the second generation AI model; The system according to Appendix 15, further including the above. (Appendix 17) One or more statements among the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt. The system according to Appendix 15. (Appendix 18) The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score based on the probability that the second generation AI model returns the correct masked key term to replace a specific masked key term included in a specific statement and the total number of masked key terms included in the specific statement, for the system described in Appendix 15. (Appendix 19) The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score, and the evaluation score is based on the total number of masked key terms included in a specific statement and the ranking of the frequency at which the correct unmasked key term used to replace a specific masked term is returned by the second generation AI model to the frequency at which the incorrect unmasked key term used to replace the specific masked term is returned by the second generation AI model, for the system described in Appendix 15. (Appendix 20) The step of determining the set of unmasked statements is executed by the second generation AI model and the third generation AI model. The step of evaluating the performance of the second generation AI model includes the step of visually representing the fairness of the second generation AI model in comparison to the fairness of the third generation AI model, for the system described in Appendix 15.

Explanation of symbols

[0073] 102 Topic 104 Role of AI 110 First generation AI model 112 Question prompt 114 Statement set 116 True statement 118 False statement 120 Manual statement input 130 Statement evaluator 140 Masking module 145 Masked statement 150 Secondary generation AI model 155 Unmasked statement 160 Evaluation module 165 Secondary generation AI evaluation< / mask> < / number> < / number> < / topic> < / airole>

Claims

1. Obtaining a topic and a role of artificial intelligence (AI) related to the topic, wherein the topic is related to a research field, and the role of the AI represents an occupational role in the research field where the topic and the role of the AI are specified by a human user; Generating a question prompt based on the topic and the role of the AI by a first generation AI model; Generating, by the first generation AI model, one or more statements as a set of statements corresponding to the question prompt; Masking key terms included in the statements of the set of statements to form a set of masked statements, wherein each statement of the set of statements includes each key term, and each masked statement included in a specific set includes at least one masked key term; Determining, by a second generation AI model, a set of unmasked statements, wherein each unmasked statement is based on a corresponding masked statement; Evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the set of statements; A method comprising the above steps.

2. Based on the evaluation of the performance of the second generation AI model indicating that the second generation AI model provides inaccurate results based on comparing the set of unmasked statements with each statement included in the set of statements, using a second training dataset different from the first training dataset initially used for training the second generation AI model to retrain or fine-tune the second generation AI model; The method according to claim 1, further comprising the above step.

3. One or more of the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt. The method according to claim 1.

4. To form the set of masked statements, the step of masking key terms included in the statements of the statement set includes: identifying one or more stop words representing common words included in the natural language processing of the statement set; excluding the one or more stop words from each statement of the statement set; identifying the key terms included in each statement based on the words remaining in each statement after excluding the one or more stop words; masking one of the identified key terms; The method according to claim 1, comprising:

5. The step of evaluating the performance of the second generation AI model includes calculating an evaluation score based on the probability that the second generation AI model returns the correct masked key term to replace a specific masked key term included in a specific statement and the total number of masked key terms included in the specific statement. The method according to claim 1.

6. The step of evaluating the performance of the second generation AI model includes calculating an evaluation score, and the evaluation score is based on the total number of masked key terms included in a specific statement and the ranking of the frequency with which the correct unmasked key term used to replace a specific masked term is returned by the second generation AI model to the frequency with which the incorrect unmasked key term used to replace the specific masked term is returned by the second generation AI model. The method according to claim 1.

7. The step of determining the set of unmasked statements is performed by the second generation AI model and the third generation AI model. The step of evaluating the performance of the second generation AI model includes visually representing the fairness of the second generation AI model in comparison with the fairness of the third generation AI model, according to the method of claim 1.

8. One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause a system to perform operations, the operations including obtaining a topic and a role of artificial intelligence (AI) related to the topic, the topic being related to a research field, and the role of the AI representing an occupational role in the research field where the topic and the role of the AI are specified by a human user; generating, by a first generation AI model, a question prompt based on the topic and the role of the AI; generating, by the first generation AI model, one or more statements as a set of statements corresponding to the question prompt; masking key terms included in the statements of the set of statements to form a set of masked statements, each statement of the set of statements including each key term, and each masked statement included in a particular set including at least one masked key term; determining, by a second generation AI model, a set of unmasked statements, each unmasked statement being based on a corresponding masked statement; evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the set of statements; One or more non-transitory computer-readable storage media including the above.

9. The operations include Evaluating the performance of the second generation AI model, based on comparing the set of unmasked statements with each statement included in the statement set, and based on indicating that the second generation AI model provides inaccurate results, using a second training dataset different from the first training dataset initially used to train the second generation AI model to retrain or fine-tune the second generation AI model, The one or more non-transitory computer-readable storage media according to claim 8, further comprising.

10. One or more of the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt. The one or more non-transitory computer-readable storage media according to claim 8.

11. To form the set of masked statements, the step of masking key terms included in the statements of the statement set is Identifying one or more stop words representing common words included in the natural language processing of the statement set; Excluding the one or more stop words from each statement of the statement set; Identifying the key terms included in each statement based on the words remaining in each statement after excluding the one or more stop words; Masking one of the identified key terms; The one or more non-transitory computer-readable storage media according to claim 8, comprising.

12. The step of evaluating the performance of the second generation AI model includes calculating an evaluation score based on the probability that the second generation AI model returns the correct masked key term to replace a specific masked key term included in a specific statement and the total number of masked key terms included in the specific statement. The one or more non-transitory computer-readable storage media according to claim 8.

13. The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score, and the evaluation score is based on the total number of masked key terms included in a specific statement, the frequency at which the correct unmasked key terms used to replace a specific masked term are returned by the second generation AI model, and the ranking of the frequency at which the incorrect unmasked key terms used to replace the specific masked term are returned by the second generation AI model. One or more non-transitory computer-readable storage media according to claim 8.

14. The step of determining the set of unmasked statements is performed by the second generation AI model and the third generation AI model. The step of evaluating the performance of the second generation AI model includes the step of visually representing the fairness of the second generation AI model in comparison to the fairness of the third generation AI model. One or more non-transitory computer-readable storage media according to claim 8.

15. A system comprising: One or more processors; One or more non-transitory computer-readable storage media configured to store instructions; Including, the instructions, in response to being executed, cause the system to perform operations, the operations including: Obtaining a topic and the role of artificial intelligence (AI) related to the topic, the topic being related to a research field, and the role of the AI representing an occupational role in the research field where the topic and the role of the AI are specified by a human user; Generating a question prompt based on the topic and the role of the AI by a first generation AI model; Generating one or more statements as a set of statements corresponding to the question prompt by the first generation AI model; Masking key terms included in the statements of the set of statements to form a set of masked statements, each statement of the set of statements including each key term, and each masked statement included in a specific set including at least one masked key term. A step of determining a set of unmasked statements by a second generation AI model, where each unmasked statement is based on a corresponding masked statement, and A step of evaluating the performance of the second generation AI model based on comparing the set of unmasked statements with each statement included in the statement set, and A system including the above.

16. The operation is Based on the evaluation of the performance of the second generation AI model indicating that the second generation AI model provides inaccurate results based on comparing the set of unmasked statements with each statement included in the statement set, using a second training dataset different from the first training dataset initially used to train the second generation AI model to retrain or fine-tune the second generation AI model, The system according to claim 15, further including the above.

17. One or more statements among the statements included in the statement set are provided by a human user and generated by the first generation AI model, and the one or more statements provided by the human user are true statements or false statements regarding the question prompt. The system according to claim 15.

18. The step of evaluating the performance of the second generation AI model includes calculating an evaluation score based on the probability that the second generation AI model returns a correct masked key term to replace a specific masked key term included in a specific statement and the total number of masked key terms included in the specific statement. The system according to claim 15.

19. The step of evaluating the performance of the second generation AI model includes the step of calculating an evaluation score, and the evaluation score is the total number of masked key terms included in a specific statement, the frequency with which the correct unmasked key terms used to replace a specific masked term are returned by the second generation AI model, and the ranking of the frequency with which the incorrect unmasked key terms used to replace the specific masked term are returned by the second generation AI model. The system according to claim 15 is based on this.

20. The step of determining the set of unmasked statements is performed by the second generation AI model and the third generation AI model. The step of evaluating the performance of the second generation AI model includes the step of visually representing the fairness of the second generation AI model in comparison with the fairness of the third generation AI model. The system according to claim 15.

Images