Techniques for evaluating bias in trained models

Abstract

A system configured to perform various bias checks on a machine learning (ML) model to identify one or more biases, if any, that may be inherent to the ML model is disclosed. Bias assessment results generated by performing the checks are then reported to users, such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. The bias assessment system performs one or more bias checks by generating a synthetic dataset using attributes present in the ML model or training dataset used to train the ML model. Prediction data is then generated by inputting synthetically generated input data points of the synthetic dataset into the ML model. The prediction data is then processed and evaluated for bias. The results of the evaluation may be compiled into a bias assessment report.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims priority to U.S. Non-Provisional Patent Application No. 17 / 508,734, entitled "TECHNIQUES FOR TRAINED MODEL BIAS ASSESSMENT", filed on October 22, 2021. The entire content of said application is incorporated herein by reference in its entirety for all purposes.

Background Art

[0002] Background In recent years, there has been a rapid increase in the adoption of artificial intelligence (AI) and machine learning (ML) solutions across many different industries and applications. For a typical ML solution, during a training stage, the ML model is trained and validated using a specific training and validation dataset. Once the model reaches an acceptable level of accuracy during the training stage, the model is then deployed to a production environment, where it is used to generate predictions for real-time production data inputs. However, if the particular training dataset used to train the ML model contains biased data, the trained ML model will also generate biased predictions.

[0003] Bias is a common, but not trivial, problem in machine learning, causing many erroneous predictions made by ML models and potentially leading to model instability in real-world scenarios. For example, a biased ML model may not only fail to predict correct information, but may also predict incorrect information that the ML model represents as a proper prediction. This creates problems for services hosting trained ML models for customers, and these problems are amplified when the service hosts untrained ML models under its supervision. Bias is not a problem that can be solved simply by increasing the size of the training dataset; in fact, a larger biased training dataset may only reinforce undesirable biased behavior in the trained ML model. Often, the trained ML model itself is a "black box" element, and bias is frequently undetectable by simply considering the organization of the trained ML model itself. [Overview of the project] [Means for solving the problem]

[0004] overview This disclosure relates to the assessment of bias in trained machine learning (ML) models. A system is disclosed configured to perform various bias checks on an ML model to identify one or more biases that may be inherent to the ML model, if any. The bias assessment results generated by performing the checks are then reported to users such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. Various embodiments, including methods, systems, non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors, and the like, are described herein.

[0005] In one embodiment, the bias assessment function is provided as a cloud service by Bias Assessment as a Service (BEaaSS). Subscribers to such a cloud service may submit ML models to the service and receive bias assessment reports generated by the service for the submitted ML models. The bias report may include information identifying bias checks performed by BEaaSS and their results. Recipients of the bias report, such as data scientists, can use the information in the bias report to reduce any identified biases. For example, a data scientist may modify the training dataset used to train the model in order to reduce a particular identified bias.

[0006] The bias assessment system embodiments described herein provide an automated solution for detecting and assessing various bias types inherent in a trained ML model. Based on the attributes of the ML model and, if possible, the potentially biased training dataset used to train the ML model, the bias assessment system is programmed to generate one or more synthetic datasets that can be input into the ML model. The trained ML model makes predictions based on the input synthetic datasets, and the predictions may be assessed for bias. The synthetic datasets include a set of synthetic data points that, when input into a trained ML model with little or no inherent training bias, do not predict biased predictive data. The same set of synthetic data points, when input into a trained ML model with substantially inherent training bias, produce substantially biased predictive data that are output. Bias may be assessed by manually examining the variability in predictions after introducing artificial bias into the dataset. As the predicted output of the ML model changes, the model may become susceptible to bias. The amount of bias may change based on how much the new outputted predictions deviate from the expected predictions.

[0007] Predictive data generated by inputting a set of synthetic data points into a trained ML model is processed by an evaluator configured to detect specific bias types within the predictive data. One or more of these evaluations may be performed for one or more specific bias types, and the results may be presented in a bias report, which may be output to the parties. In some cases, the results and / or report may be used to perform a number of downstream actions. The results, report, and actions provide a bias detection and evaluation mechanism through which the ML model host, users, and other parties can improve model training, testing, and execution.

[0008] Trained ML models to be evaluated for bias according to the embodiments described herein may be received from a number of sources (e.g., user / customer devices, repositories or trained ML models, serverless functions, etc.). In some embodiments, a customer sends a trained ML model to a Bias-as-a-Service System (BEaaSS) to determine whether the customer model generates predictive data using one or more biases formed from it. In some embodiments, a model catalog system is configured to receive trained ML models to store in a model catalog. The model catalog system may communicate with BEaaSS to determine whether the received model generated predictions according to an acceptable level of bias before including the model in the model catalog.

[0009] In some embodiments, the bias evaluation system implements an attribute classifier for determining one or more attributes of the trained ML model and / or the training data used to train the ML model. The attribute classifier is configured to process the ML model data and / or the training data used to train the ML model in order to determine one or more attributes of the ML model. The bias evaluation system may use one or more determined attributes to generate synthetic data that may be input into the trained ML model to generate predictions.

[0010] A synthetic data generator may generate synthetic data based on one or more determined attributes of a trained ML model. One or more synthetic datasets may be input to the trained ML model to generate predictive data. The predictive data may be parsed by one or more checking entities in BEaaSS to determine whether the predictive data corresponds to a type of bias. One or more synthetic datasets may be generated in some form to attempt to influence the trained ML model to output biased predictive data. For example, a synthetic dataset may be generated with an unbalanced distribution of attribute values ​​at various synthetic data points to attempt to influence the trained ML model to generate and output biased predictive data. In some embodiments, synthetic datasets may be stored in a synthetic data repository after generation. If a request is received to evaluate the trained ML model for bias, and the attributes of the trained ML model share the same attributes as synthetic datasets stored in the repository, the synthetic datasets may be selected for input to the trained ML model without requesting the generation of new synthetic datasets.

[0011] A bias evaluation system may be programmed to perform a number of bias checks in an automated manner, each bias check being for bias in the predicted data output by an ML model given one or more synthetic datasets as input. Each bias check corresponds to a different bias type that may be specific to the trained ML model. There are no restrictions on the types of bias that the bias evaluation system can evaluate for a trained ML model, and the evaluation may be performed in any combination or form that is useful in evaluating the ML model.

[0012] A bias evaluation system uses a synthetic dataset input to determine whether the trained ML model behaves according to one or more biases, and utilizes the predicted data generated by the trained ML model. In some embodiments, the bias evaluation system may supplement its evaluation of the predicted data with an additional evaluation of the training dataset used to train the ML model. For example, a bias evaluation entity in the bias evaluation system may combine the results of bias checks on the predicted data with the results of bias checks on the training data to generate combined result data.

[0013] The results obtained by the bias evaluation system by performing a bias check may then be output along with the predictions made using the ML model. These results provide consumers of predictions with additional information regarding the probability that the predictions made by the ML model are biased and the corresponding potential impact on the generated predictions. Thus, the bias evaluation results generated and output by the bias evaluation system can provide a warning system for when the model's predictions are biased, when the model itself is improperly trained, or when the training dataset used to train the ML model is flawed.

[0014] The results of one or more bias assessments for one or more bias types may be compiled and used by a bias assessment system to generate a bias report. The bias report may include reporting information indicating the tendencies of the trained ML model to generate predictive data corresponding to one or more specific biases. The bias report may include reporting information related to one or more bias checks performed and the corresponding bias check results. The bias report may further include an overall bias score indicating the total degree of bias detection in the predictive data and / or one or more downstream actions or recommendations that may be taken as a result of the bias assessment.

[0015] A set of one or more bias check results may be used to determine one or more downstream actions to perform in response to the generation of one or more bias check results. For example, one downstream action may be to generate a message, based on one or more bias check results, that the trained ML model has a tendency to generate biased predictive data and / or that the trained ML model was trained using a flawed / biased training dataset, and send this message to the company's data scientists. Another downstream action may be to prevent further proliferation of the trained ML model or to prevent the trained ML model from being included in a model catalog. Yet another downstream action may be to send the predictive data and bias report to the client / service that originally supplied the trained ML model.

[0016] In some embodiments, the bias evaluation system may be provided as a cloud service by a cloud service provider. The service is made available to customers or subscribers who subscribe to this service and other services provided by the cloud service provider.

[0017] In one embodiment, a technique is disclosed in which a bias evaluator as a service system performs a process comprising: a computing system determining a set of model attributes for a trained model to be evaluated; the computing system generating a first synthetic dataset used for a first bias check performed for the trained model based on the set of model attributes, the first bias check being configured to evaluate the trained model with respect to a first bias type, the first synthetic dataset comprising a plurality of data points; and further comprising using the trained model to generate first predictive data for the first synthetic dataset, the first predictive data comprising a plurality of first predictive values ​​generated by the trained model for the plurality of data points in the first synthetic dataset; the computing system generating a first bias result for a first bias type based on the first predictive data; and the computing system generating a bias evaluation report for the trained model, the bias evaluation report comprising information indicating the first bias result.

[0018] In one embodiment, the first bias result includes one or more bias values ​​generated based on first prediction data. In some further embodiments, the bias evaluation report includes the first bias result and one or more bias values, and the method further includes outputting the bias evaluation report. In other further embodiments, the method further includes comparing at least one or more bias values ​​with a bias-to-bias threshold, and based on the comparison, deciding whether to accept or reject the trained model from the group of trained models.

[0019] In one embodiment, the process further comprises the computing system generating a second synthetic dataset used for a second bias check performed for a trained model, based on a set of model attributes, the second bias check being configured to evaluate the trained model with respect to a second bias type, the second synthetic dataset containing multiple data points, and using the trained model to generate a second set of predictions for the second synthetic dataset, and further comprising the computing system generating a second bias result for a first bias type based on the first prediction data. In some further embodiments, the method further comprises the computing system generating bias scores based on the first and second bias results, and deciding, based on the generated bias scores, whether to accept or reject the trained model from being included in a group of trained models.

[0020] In some embodiments, determining a set of model attributes involves processing a trained model to determine at least one model attribute in the set of model attributes. In some embodiments, determining a set of model attributes involves determining at least one model attribute in the set of model attributes based on an analysis of the training data used to train and generate the trained model.

[0021] In one embodiment, the process further includes determining the training data used to train and generate a trained model, the computing system further includes generating a second bias result for a first bias type based on the training data, and generating the first bias result further based on the generated second bias result. In some embodiments, generating a first synthetic dataset includes the computing system generating a first synthetic dataset using a generative neural network machine learning model based on a set of model attributes for a trained model.

[0022] In one embodiment, a system such as a bias estimator as a service system includes a processor, and a memory including instructions that, when executed by the processor, cause a device to perform the processes described herein. In another exemplary embodiment, a non-transitory computer-readable medium stores a plurality of instructions executable by one or more processors to cause the one or more processors to perform the processes described herein.

[0023] The above features and aspects, along with other features and aspects, will become more apparent upon reference to the following specification, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] [Figure 1] FIG. 1 is a simplified diagram of a distributed environment incorporating a bias estimator as a service system in accordance with various embodiments. [Figure 2] FIG. 2 is a simplified diagram of a bias checker component of a bias estimator as a service system in accordance with various embodiments. [Figure 3] FIG. 3 is a simplified diagram of an attribute identifier component of a bias estimator as a service system in accordance with various embodiments. [Figure 4] FIG. 4 is a simplified flow diagram illustrating an exemplary process for generating a bias evaluation for a trained machine learning model using a bias estimator as a service system in accordance with various embodiments. [Figure 5] FIG. 5 is a diagram illustrating an exemplary bias evaluation report generated by a bias estimator as a service system in accordance with various embodiments. [Figure 6] FIG. 6 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 7]This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 8] This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 9] This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 10] A block diagram showing an exemplary computer system according to at least one embodiment. [Modes for carrying out the invention]

[0025] Detailed explanation In the following descriptions, certain details are given for illustrative purposes to provide a complete understanding of certain embodiments. However, it will become clear that various embodiments can be performed without these specific details. The drawings and descriptions are not intended to be restrictive. The word “exemplary” is used herein to mean “acting as an example, illustration, or reference.” Any embodiment or design described herein “exemplary” should not necessarily be construed as being preferable or advantageous to other embodiments or designs.

[0026] This disclosure relates to the assessment of bias in trained machine learning (ML) models. A system is disclosed configured to perform various bias checks on an ML model to identify one or more biases that may be inherent to the ML model, if any. The bias assessment results generated by performing the checks are then reported to users such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. Various embodiments, including methods, systems, non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors, and the like, are described herein.

[0027] In one embodiment, the bias assessment function is provided as a cloud service by Bias Assessment as a Service (BEaaSS). Subscribers to such a cloud service may submit ML models to the service and receive bias assessment reports generated by the service for the submitted ML models. The bias report may include information identifying bias checks performed by BEaaSS and their results. Recipients of the bias report, such as data scientists, may use the information in the bias report to reduce any identified biases. For example, a data scientist may modify the training dataset used to train the model in order to reduce a particular identified bias.

[0028] There are various reasons why bias can creep into a trained ML model. Typically, this occurs when the dataset used to train the model contains data points that are heavily represented or weighted towards certain attributes. As a result of such a weighted or skewed dataset, the trained model may not adequately represent the real-world environment or use case that the trained model is used to make predictions. For example, a model trained to select resumes of potential candidates may be biased towards selecting male or female resumes if the training dataset used to train the model mainly contains male resumes with very few data points corresponding to female resumes. Trained ML can be biased by different types of bias in the training dataset, including sample bias, exclusion bias, measurement bias, label / recall bias, observer bias, association bias, gender bias, religious bias, racial bias, and others. High levels of bias can cause the model to overlook relevant relationships between data features or attributes and the outputs predicted by the model.

[0029] If the specific training dataset used to train an ML model contains biased data, the trained ML model is also more likely to make biased predictions. Bias is a common but not trivial issue in ML and is the cause of many erroneous predictions made by ML models. For example, a biased ML model may not only fail to predict correct information, but may also predict incorrect information that the ML model represents as an appropriate prediction. For instance, a trained ML model that generates predictive data in a biased form may be more likely to produce an erroneous prediction when certain input data points are provided to the machine learning model. A trained ML model is "biased" when the predictive data it generates contains undesirable values ​​or results that would not appear in the same proportion if generated by a properly trained ML model. Bias arises when an ML model is trained in a particular form so that it tends to output biased predictive data. Because training datasets are often non-uniform (i.e., the attributes of the data in the dataset used to train the model are often incomplete or biased), trained ML models are often biased by training on these datasets. For example, a "biased" ML model may tend to predict incorrect values ​​or values ​​outside the expected range of the base given the input data points. A biased model will also alter the output predictions in a way that is heavily influenced by the input data, along with a high level of bias in the input dataset. Bias can manifest in many types and forms, including, for example, selection bias, stereotyping bias, reporting bias, in-group bias, and anecdotal fallacy.

[0030] Biased ML models produce undesirable predictions more frequently than well-trained ML models. Even worse, a model may be biased towards a particular prediction for one attribute / type rather than for others. Thus, a model may appear to predict data with high accuracy for some input data points, but produce biased predictions for other, more specific input data points. This creates a problem for services hosting trained ML models for customers, and the problem is even greater when a service hosts untrained ML models under the service provider's supervision. For example, in the case of a service provider hosting ML models for customers where the service provider is not involved in training the model and has no control over its training, the ML model is like a "black box" to that service provider, which may have no insight into the potential biases of the ML model. Bias is not a problem that can be solved simply by increasing the size of the training dataset; a larger, biased training dataset may only amplify undesirable biased behavior in the trained ML model. The "black box" nature of ML models means that bias often cannot be detected by simply examining the organization of the trained ML model itself.

[0031] The bias evaluation system embodiments described herein provide an automated solution for detecting and evaluating various bias types that may be inherent in a trained ML model. In one embodiment, the input provided to the bias evaluation system for the ML model to be evaluated may be the model to be evaluated, and, if available, may be a training dataset used to train the ML model. The bias evaluation system is programmed to determine the attributes of the ML model to be evaluated. Once the training dataset is provided as input, the bias evaluation system may analyze the training dataset to determine the attributes of the input to the model. Based on the identified attributes, the bias evaluation system uses synthetic data generation techniques to generate one or more synthetic datasets, each synthetic dataset directed to check for a particular bias and containing a number of synthetically generated data points to check for that particular bias.

[0032] The bias assessment system then performs various bias checks using the synthetic dataset and generates assessment results. For a particular bias to be assessed, the bias assessment system inputs the synthetic dataset generated for that bias into the ML model to be assessed and determines the predicted values ​​generated by the ML for each input data point from the synthetic dataset. The predicted values ​​generated by the ML model for the data points in the synthetic dataset for a particular bias check are collectively called the predicted data generated by the ML model for the synthetic dataset for that particular bias check. The bias assessment system then analyzes the predicted data to generate a bias assessment result for that particular bias check. In one embodiment, the bias assessment result indicates the degree of bias shown by the ML model for the particular bias to be assessed. The degree may be shown by quantification, scores, visualizations (e.g., graphs), and similar.

[0033] An example of such a ML model is a trained neural network (NN) consisting of many nodes organized into layers, each layer containing one or more nodes. The layers include an input layer to which inputs are provided, and an output layer that outputs predicted values ​​for the inputs provided as inputs to the input layer. There may be zero or more hidden layers between the input and output layers, and the output of one layer is provided as input to the next layer. In such a neural network model, for synthetic data points provided as inputs to the input layer of the neural network, the output layer of the neural network outputs predicted values ​​representing the values ​​predicted by the neural network for the inputs.

[0034] In addition, as part of generating predicted values, each of the hidden layers and the nodes within the layer may generate intermediate values, which are passed to the next layer until they reach the output layer and the output layer outputs predicted values. In such embodiments, the predicted values ​​generated by the model for the synthetic dataset include the final values ​​predicted by the output layer of the neural network. This predicted data is then analyzed by a bias evaluation system to generate bias results for specific bias checks. As described herein, the final result of an ML model processing an input is referred to as “predicted data”.

[0035] For example, once prediction data is generated, the prediction data is processed by an evaluator configured to detect specific bias types within the prediction data. The evaluation may use the prediction data for a synthetic dataset to evaluate an ML model for a particular bias, and may also use other aspects of the bias detection process, such as one or more attributes, input synthetic data points, etc. For example, the bias evaluator may compare the prediction data output by the trained ML model with one or more synthetic input values ​​to determine the expected distribution of the predictions and the actual distribution of the predictions. In some embodiments, the evaluation may be based on statistical decisions and functions applied to the set of prediction data.

[0036] One or more of the evaluations may be performed for one or more specific bias types, and the results may be presented in a bias report. The report may be output to a party. For example, for a trained ML model being evaluated, the results of various bias checks performed for that model may be compiled or aggregated into a comprehensive report reviewable by the submitting entity that submitted the trained ML model for bias evaluation. The report may include information indicating the various bias checks performed by the bias evaluation system and their corresponding evaluation results, an overall bias evaluation result, and a number of proposed downstream actions, such as sending the report to an entity, rejecting the model from the model catalog, or retraining the ML model. In some embodiments, a user or entity receiving a bias evaluation report may select one or more downstream actions to perform based on the generated bias report. For example, the service performing the report generation may also include one or more subsystems, thereby indicating to the user that one or more subsystems should be activated. For example, a retraining subsystem for retraining a biased model may be suggested by the bias report, and the user may indicate to the subsystem to retrain the biased model in response to receiving the report, etc.

[0037] In one embodiment, the degree of a particular bias may be represented using a quantification associated with the bias result. Various different quantifications, such as scores and graphs, may be used to represent the degree of bias. For example, the results of various bias checks performed may be compiled into a number of visual graphs and score quantifications that are recognizable and understandable by humans and / or machines.

[0038] In some embodiments, an overall bias score may be displayed in the bias assessment report. The overall bias assessment may be an edited bias assessment score based on a number of subscores corresponding to a number of specific biases being assessed. For example, the overall bias assessment score may be an inclusive score such as an average score, or multiple subscores of individual bias assessments.

[0039] In some cases, results and / or reports may be used to perform a number of downstream actions. Downstream actions may include, for example, corrective actions to correct bias, or other response actions in response to the results of a bias assessment. Corrective actions may be actions that correct biased behavior in the model in response to the detection of bias in the results. For example, retraining an ML model with an unbiased training dataset may be a downstream action that corrects biased behavior in an ML model. Corrective actions may be initiated by the user in response to the receipt of a bias report. Response actions may be actions that occur based on the state of the model after assessment. For example, decisions to refuse to include a model in a model catalog and rejecting a model from a model catalog may occur in response to a decision that an ML model is operating with an unacceptable level of bias.

[0040] As a result, the reports and actions provide a bias detection and evaluation mechanism through which ML model hosts, users, and other parties can improve model training, testing, and implementation. For example, in some embodiments, a model catalog that provides a repository for ML models submitted by users may use BEaaSS's bias evaluation service. The model catalog may submit a bias evaluation request to BEaaSS, which may include the ML model to be evaluated and the training data used to train the model. After performing the bias evaluation, BEaaSS may generate a bias evaluation report and send the report back to the model catalog in response to the request sent by the model catalog. The model catalog may then perform one or more actions based on the bias evaluation report. In one implementation, if the report indicates that an ML model has failed the bias evaluation, the model catalog may refuse to add the model to the model catalog. The model catalog may only allow an ML model to be added to the model catalog if it receives a passing bias evaluation result from BEaaSS.

[0041] The technologies for facilitating BEaaSS offer numerous technical advantages that improve the functionality and implementation of trained ML models. For example, the BEaaSS described herein provides novel automated bias assessment systems and services that were previously unavailable. For instance, model assessment is primarily performed manually and bias by bias. Implementing BEaaSS as described herein enables automated, comprehensive bias assessment for trained ML models. BEaaSS facilitates the technology to automatically ingest trained ML models in response to requests, perform comprehensive bias assessments, and output results. This provides a centralized system for automated model improvement for various bias tests.

[0042] The described BEaaSS also offers technical advantages by removing work from clients and owners of trained ML models. Due to BEaaSS's automated nature, model owners / model catalogs do not need to actively participate in model evaluation and bias assessment. Checks, including the generation of synthetic datasets for performing the checks, are performed automatically by BEaaSS without requiring input from the client beyond the request to evaluate the model. This improves user resource utilization while centralizing the system for evaluating ML models. Users do not need to know what checks are being performed or the central mechanism for the checks. Instead, users simply generate a request for model evaluation and wait for the automatically generated report in response to the request.

[0043] The described BEaaSS also offers flexible checking capabilities that can be updated as needed as a centralized service. For example, as new bias checks progress, improve, and otherwise correct, the aggregated nature of BEaaSS allows users to update checks in one place without having to manually download and add the latest set of bias checks. This makes BEaaSS easier to integrate into user workflows and also saves user resources. This also ensures that secure applications such as model catalogs are always updated with the most modern checks to comprehensively diagnose new problematic biases in near real-time.

[0044] Exemplary Systems and Embodiments Figure 1 is a simplified diagram of a distributed environment incorporating a bias evaluator as a service system in various embodiments. As shown in Figure 1, the distributed environment includes numerous systems and subsystems. The distributed environment includes Bias Evaluation as a Service System (BEaaSS) 100. BEaaSS 100 may be a service implemented in a computing device such as a server system. BEaaSS may be a service configured to facilitate the embodiments described herein, namely, the evaluation of a trained ML model for one or more bias types. For example, BEaaSS may be a service configured to receive a trained ML model from a separate external system, evaluate the trained ML model for bias, return a bias report based on the evaluation, and / or perform downstream operations.

[0045] The controller subsystem 130 is a subsystem of the BEaaSS 100. The controller subsystem 130 may be configured to facilitate the operation of the BEaaSS 100 and its communication with external systems. For example, the controller subsystem 130 may be a central subsystem having access to various other subsystems of the BEaaSS 100. The controller subsystem 130 may therefore facilitate the transmission of data within the BEaaSS 100 and invoke various other subsystems to perform the embodiments described herein. The controller subsystem 130 may communicate with systems outside the BEaaSS 100 to take in and output data as part of the embodiments described herein. For example, the BEaaSS 100 may be implemented as part of a model evaluation service provided to a customer by a service provider. The controller subsystem 130 may be configured to communicate with a customer / user device to receive a trained ML model for evaluation as part of the service.

[0046] The user device 110 may be a computing device or system that is communicably coupled to the BEaaSS 100, for example, via a controller subsystem 130. The user device 110 may be a computing device used by a customer of a service implementing the BEaaSS 100 to submit a trained ML model for evaluation. For example, a service provider may provide a service implementing the BEaaSS 100, through which a customer / user may submit a trained ML model owned by the customer / user for evaluation. In response to submitting the trained ML model to the BEaaSS 100, the customer may receive a response containing information and / or behavior related to the trained ML model.

[0047] The user device 110 may send a model, training data, and bias type 111 for evaluation to the controller subsystem 130. The model sent to the controller subsystem 130 is a trained ML model to be evaluated for bias. In some embodiments, the user device may also send training data to the controller subsystem 130. The training data may include at least a training dataset used to train the trained ML model, which is also sent to the controller subsystem. In some embodiments, a bias type may also be sent to the controller subsystem 130. The bias type may refer to one or more bias checks for which a customer using the user device 110 may request part of the bias evaluation for the trained ML model. In some embodiments where the bias type is not sent, all available bias checks are performed on the trained ML model by default. In some embodiments where the bias type is not sent, a subsystem of BEaaSS 100, such as an attribute classifier, may process the trained ML model and optionally the training dataset to determine one or more bias types that are possible or likely to be specific to the trained ML model.

[0048] Upon completion, a bias assessment report 112 may be sent from the controller subsystem 130 to the user device 110 in response to the transmission of the model and training data and bias type 111 for evaluation. Further information about the bias assessment report is described below.

[0049] In some embodiments, BEaaSS100 is communicatively coupled to a model catalog 120. The model catalog 120 may be a model service that hosts a number of trained ML models that may be used by customers of the model service. For example, the model service may allow source customers / users to upload their own trained ML models to be included in the model service. Other customers / users may then utilize the source customer / user's trained ML models. As described above, for this reason, it is important for the model service provider to host only trained ML models with little or no inherent bias in order to maintain an optimal model service.

[0050] In some embodiments, the model catalog 120 may also include a domain ontology store for storing multiple models, where each model is grouped into a domain. Each domain specifies attributes or industries common to multiple models. The model recommendation engine of the model catalog 120 can compare customer-provided data with multiple models in the domain ontology store to identify one or more recommended models for customer use. For example, the model recommendation engine can identify a number of similar terms between the parsed data and terms associated with each model. In some examples, the model recommendation engine can generate confidence quantifications for each model. The confidence quantifications may include a value (a percentage from a range of values) indicating the estimated likelihood that a model is relevant to the set of data provided by the customer. The confidence value may be based on the number of common terms between each model and the parsed data from the dataset. The model recommendation engine can identify one or more specific recommended models for the data provided by the customer. For example, a list of recommended models and a description of each recommended model may be provided to the client device for selecting a recommended model. The model catalog 120 can generate a domain ontology store and continuously add new models to the domain ontology store.

[0051] The feedback and learning engine can obtain feedback data from model recommendations and subsequent selection / exclusion of models for specific sets of data provided by the customer. As mentioned above, it is important that the models provided to the customer generate predictive data with little to no bias. Maintaining only models in the model catalog that contain an acceptable level of bias is a very important goal for model catalog service providers.

[0052] Model catalog 120 may receive models 121 for submission and training data. Models 121 for submission and training data may be received, for example, from a model catalog user who is submitting a model for inclusion in the model catalog. Training data may also be supplied along with the model for submission. Model catalog 120 is communicably coupled to BEaaSS 100 via controller subsystem 130. Model catalog 120 may send a request to BEaaSS 100 to perform an evaluation of the model for submission before accepting the model for submission as part of model catalog 120.

[0053] The model catalog 120 may send the model 122 for submission and training data and bias types to the controller subsystem 130. The model for submission and training data is the same data received as part of the submission described above. In addition, the model catalog may generate bias types corresponding to bias checks that the model catalog requires the model for submission to pass before it is accepted into the model catalog, and send these to BEaaSS 100. The bias types generated by the model catalog 120 may correspond to several bias checks that are important to a particular model catalog, while excluding others that are not important. For example, a model catalog configured to host models for predicting employee salaries may require that the gender, race, and language biases being evaluated are at levels below thresholds before the models can be included in them. Other biases, such as foul language and social media presence, may not be important factors for the models in the model catalog and are therefore not included in the bias types generated and sent from the model catalog 120.

[0054] Upon completion, the bias evaluation report 123 may be sent from the controller subsystem 130 to the user device 110. The bias evaluation report 123 may be similar to the bias evaluation report 112 that may be sent to the user device 110 as described above. In addition, the bias evaluation report 123 may include one or more sets of data corresponding to recommended decisions that the model catalog 120 may make regarding the submitted model. For example, the bias evaluation report may include recommendations that the model catalog 120 accept or reject models for submission to be included in the model catalog 120.

[0055] The customer bias evaluation preference 131 is one or more sets of bias preference configurations stored in the controller subsystem 130. In embodiments where bias types corresponding to bias checks to be performed are not sent to the BEaaSS 100, the customer bias evaluation preference 131 may include data on one or more preferred bias types for evaluating a trained ML model. For example, a user of user device 110 may be associated with a customer bias evaluation preference that enumerates one or more bias types that the user typically uses in trained model bias evaluation. In other examples, the customer bias evaluation preference 131 stores information on optimal bias types related to the model catalog 120. As described above, bias types may determine one or more bias checks to perform for a given trained ML model. Once the trained ML model and bias types are determined / received, the BEaaSS 100 may begin by evaluating the trained ML model for bias.

[0056] BEaaSS100 may first determine one or more attributes to generate a synthetic dataset that can be input to a trained ML model. The controller subsystem 130 sends the model 131 for evaluation and training data to the attribute classifier 140. The attribute classifier 140 processes the model 131 for evaluation and training data and may determine one or more attributes to generate a synthetic dataset. The function of the attribute classifier 140 is described below with reference to Figure 3. Once the attributes are determined, the attribute classifier 140 sends the attributes 141 to the controller subsystem 130. In some embodiments, the attribute classifier 140 may directly send one or more attributes 141 to a synthetic data generator, such as a synthetic data generator 150.

[0057] Once one or more attributes for generating synthetic data are determined, BEaaSS100 generates the synthetic data, which may be input to a trained ML model to generate predictive data and output. The controller subsystem may transfer the attribute 141 received from the attribute discriminator 140 to the synthetic data generator 150. The synthetic data generator 150 may be a subsystem BEaaSS100 configured to generate one or more synthetic datasets, which may be input to a trained ML model, thereby causing the trained ML model to generate and output predictive data. The synthesis may be generated such that, when the synthetic dataset is input to one or more trained ML models, one or more trained ML models are most likely to generate predictive data corresponding to biased predictions. More specifically, the synthetic input data points in the synthetic dataset are generated such that the input data points contain values ​​that, when input to a trained ML model, generate predictive data that more clearly demonstrates the potential biases inherent in the trained ML model.

[0058] The synthetic data generator 150 may employ one or more ML techniques for generating a synthetic dataset. These techniques may include one or more machine learning (ML) techniques, rule-based techniques, and others. In some implementations, one or more machine learning-based techniques may be used. For example, a generative adversarial network (GAN) may be used to generate synthetic data in a synthetic dataset, in which case the generated synthetic data closely resembles the original or actual data. An example of a GAN architecture is described in "Ian J. Goodfellow et al., Generative Adversarial Nets, NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, Volume 2, December 2014, pp. 2672-2680". The entire contents of Goodfellow et al.'s publication are incorporated herein by reference for all purposes. In some embodiments, generative models such as natural language generation models (NLGs) and hidden Markov models (HMMs) and artificial neural networks (ANNs) are used.

[0059] GANs can generate synthetic data based on real data provided as input to the GAN. The synthetic data generated by a GAN resembles the real data in terms of essential parameters, univariate and multivariate distributions, and interrelationships between variables. During training, the GAN learns the true data distribution of the input training dataset, not simply replaying the old data on which the model was trained, but generating new data points from this distribution with some modifications. In a particular use case, the synthetic data generated by a GAN may be used to augment real data and generate a synthetic dataset. Since a GAN may generate data from the distribution of the training data on which the GAN was trained, one or more GANs may be trained for well-known biases, thereby also generating datasets with those biases. Thus, one or more GANs may be modified to output biased datasets in a controlled format for use when testing a model for biases. Deep learning models can also modify existing datasets to generate modified synthetic datasets containing specified biases, as part of the bias detection techniques described herein.

[0060] A typical GAN ​​architecture consists of two adversarial models, commonly implemented as neural networks competing against each other. These adversarial models include a generator neural network (generator) and a discriminator neural network (discriminator). The generator is trained to generate new synthetic data based on real data provided as input to the generator. The discriminator is a type of classifier trained to distinguish between real data and synthetic data by estimating the probability that a sample generated by the generator is real data or generated data. During GAN training, the generator and discriminator play a continuous adversarial game, and as training progresses, the generator learns to generate more realistic data samples based on the input training data, and the discriminator learns to improve at distinguishing generated synthetic data from real data. This adversarial cooperation between the two networks is the source of GAN success, as they learn from each other at the expense of one another and reach equilibrium over time.

[0061] A trained GAN can then be used to generate synthetic data for data provided as input to the GAN. The GAN may be used, for example, to generate the various synthetic datasets described in this disclosure. Different GAN architectures exist for generating different types of synthetic data, including architectures for generating synthetic tabular data. The GAN may be used to introduce heterogeneity into the generated synthetic datasets. For example, in an attempt to "trick" a potentially biased ML model, the GAN may generate a synthetic dataset for a heterogeneous bias test (i.e., the synthetic dataset has a different structure of data attributes, including missing values, missing data types, etc.). This allows for the introduction of data attributes or data ranges that may not have been present in the training dataset used to train the ML model.

[0062] Other machine learning-based techniques besides GANs may be used to generate the synthetic data described in this disclosure. These techniques may include using neural networks (e.g., convolutional neural networks (CNNs)), variational autoencoders (VAEs), decision trees, random forest techniques, linear regression, other deep learning techniques, and others.

[0063] Additionally, non-machine learning-based techniques may be used in addition to or instead of machine learning-based techniques to generate synthetic datasets. These include, for example, various sampling and best-fit techniques, Monte Carlo techniques, and others.

[0064] The synthetic data repository 151 may be a repository within the synthetic data generator 150 configured to store the generated synthetic datasets. Once a synthetic dataset is generated by the synthetic data generator 150, the synthetic dataset may be transferred to another subsystem of BEaaSS 100 and / or stored in the synthetic data repository 151. Storing the generated synthetic dataset in the synthetic data repository 151 may include storing the synthetic dataset according to a specific bias associated with the generated synthetic dataset. For example, the synthetic data generator 150 may receive an attribute 141 and a list of bias types, including “Bias A” which is performed using the generated synthetic data. The synthetic data generator 150 may generate a synthetic dataset with one or more attributes for use in evaluating a model for bias A. The synthetic dataset for bias A may then be stored in the synthetic data repository 151.

[0065] In some embodiments, synthetic datasets may be retrieved from the synthetic data repository 151 instead of, or in addition to, the generation of new synthetic data. For example, in response to receiving one or more attributes 141 and a list of bias types, the synthetic data generator may search the synthetic data repository 151 for previously generated synthetic datasets corresponding to a particular bias type and one or more attributes. Once a stored synthetic dataset is found, it may be retrieved and sent to another subsystem of BEaaSS 100. In some embodiments, techniques such as k-partition verification are used to improve the test quality of synthetic dataset generation.

[0066] The synthetic data generator 150 may send the synthetic dataset 152 to the controller subsystem 130 and / or the bias checker subsystem 160. In some embodiments, the synthetic data generator 150 sends the generated synthetic dataset 152 directly to a bias checker subsystem, such as the bias checker subsystem 160. The synthetic dataset 152 may be used in a responsive manner by the bias checker subsystem, which is to evaluate the performance of the trained ML model. In some embodiments, the synthetic data generator 150 sends the generated synthetic dataset 152 to the controller subsystem 130. The controller subsystem 130 may then group the generated synthetic dataset in a responsive manner into packages of data that can be used to perform an evaluation of the trained ML model. For example, the controller subsystem 130 may group the received synthetic dataset 152 together with the trained ML model for evaluation, the training data used to train the trained ML model, and bias type data specifying one or more bias checks to perform. The data may be grouped as a set of check data 132 sent to a bias checker subsystem 160 to perform one or more bias checks for a trained ML model. The check data 132 may be a comprehensive set of data including one or more synthetic datasets 152, instructions for one or more bias checks to perform, and originating entities for requests to perform bias assessments.

[0067] The bias checker 160 may be a subsystem of BEaaSS100 that utilizes one or more bias checkers 161(A)~(N) to perform bias evaluations for various bias types. The bias checker 160 may be configured to receive check data 132 from the controller subsystem 130 and / or synthetic dataset 152 from the synthetic data generator 150 to initiate the execution of bias evaluations for a trained ML model. In various embodiments, the bias checker subsystem 160 receives a list of bias types and parses the list to determine one or more bias checks to be performed by one or more bias checkers 161. The bias checker subsystem 160 may transfer data such as the trained ML model to be evaluated, synthetic data, and optionally the training dataset to one or more bias checkers 161 to constitute an evaluation of the trained ML model.

[0068] Bias checkers 161(A)-(N) are configured to receive data containing a synthetic dataset and data containing a trained ML model for evaluation, and in response generate and output result data 162(A)-(N). Result data 162(A)-(N) correspond to the results of bias evaluation performed on the trained ML model. Further information regarding the functionality of bias checker 161 is described below with reference to Figure 2.

[0069] The result data 162(A)-(N) generated by the bias checkers 161(A)-(N) are sent to one or more subsystems of BEaaSS 100. For example, the result data 162(A)-(N) are sent to a report generator 170. The report generator 170 may be a subsystem of BEaaSS 100 configured to generate one or more reports containing the result data 162(A)-(N) or other information related to the results of the bias assessment. In some embodiments, the report information is compiled into a human and / or computer-readable format for processing and determining the aspects of the bias assessment. In some embodiments, the overall score generator 171 is a subsystem of the report generator 170 configured to derive a single score for the bias assessment process. For example, the overall score generator 171 may utilize the result data 162(A)-(N) to generate an overall score for bias with respect to the trained ML model being evaluated. The reports generated by the report generator 170, and in some embodiments the overall scores generated by the overall score generator 171, are sent to separate entities in the report data 172. For example, the report data 172 may be returned to the user device 110 along with the bias evaluation report 112, or the report data 172 may be returned to the model catalog 120 as part of the bias evaluation report 123. Further details regarding the bias evaluation report are described below with reference to Figure 5.

[0070] The result data 162(A)-(N) may also be sent to the operation subsystem 180. The operation subsystem 180 may be a subsystem of BEaaSS100 configured to take the result data 162(A)-(N) and, in response, produce the execution of one or more actions or generate recommendations for the execution of one or more actions. The action configuration 181 may be a set of configuration data that specifies how actions are performed based on the result data 162(A)-(N). The action configuration 181 may specify a configuration for which actions are performed when the bias result or bias score meets a quantified threshold for an action. The operation subsystem 180 may receive the overall score 173 from the overall score generator 171. The overall score 173 may be parsed and compared to an action threshold to determine whether an action should be performed based on the overall score 173. For example, the action configuration 181 may specify that the operation subsystem 180 should send a report to a data scientist if the overall score 173 exceeds a pre-configured bias threshold for reporting. In some configurations, the model acceptance rules 182 are a set of rules within the operational subsystem 180. The model acceptance rules 182 specifically describe the criteria for accepting or rejecting models from the model catalog 120. For example, the model acceptance rules may specify that the operational subsystem 180 should send a recommendation to the model catalog 120, along with a bias assessment report 123, to accept the trained ML model if the bias detected for model prediction is below an acceptable bias threshold. The operational subsystem 180 may then perform operation 183.

[0071] Further details related to the processes performed by BEaaSS100 for generating bias assessment reports provided with predictive data are described below with reference to Figures 2, 3, 4, and 5.

[0072] Figure 2 shows a simplified diagram of the bias checker component of a bias evaluator as a service system in various embodiments. As shown in Figure 2, the shown bias checker includes several components through which data is generated. In particular, Figure 2 shows the bias(A) checker 161(A) for acquiring synthetic data and outputting bias evaluation results.

[0073] As part of the bias checker 160, synthetic data 200 for bias (A) is received by the trained model 210 being evaluated. The model 210 being evaluated is a trained ML model whose evaluation is sought. In some embodiments, the synthetic data for bias (A) is a set of synthetic data specifically generated to perform a bias evaluation for "bias A" in a particular trained model 210 being evaluated, and is generated by the synthetic data generator 150 for that particular purpose. The trained model 210 being evaluated is configured to receive the synthetic data 200 for bias (A) as input and to generate and output predictive data 220 for bias (A). The predictive data 220 for bias (A) may be predictive data containing one or more predictive values ​​generated by the trained model 210 being evaluated using an NN ML model.

[0074] In an exemplary embodiment, bias A is a gender bias, and the bias test for bias A is a bias test for gender bias in the trained ML model. In the same exemplary embodiment, the trained model 210 being evaluated is a model for predicting employee salaries given input data points containing attribute values ​​such as employee name, gender, and job title. In the same exemplary embodiment, the synthetic data 200 for bias (A) is a synthetic dataset containing several synthetic input data points along with attribute values ​​corresponding to theoretical employee name, gender, job title, etc. Thus, when the synthetic data 200 for bias (A) is input to the trained model 210 being evaluated, the trained model outputs predictive data for bias (A) corresponding to several salary predictions for each of the synthetic input data points. In the same exemplary embodiment, the synthetic data 200 for bias (A) may be generated in a form that indicates bias or bias (A) 220 in the predictive data. For example, two different input data points may contain similar values ​​across the data points, with the sole exclusion of values ​​for gender. If dissimilar predictive results are generated for both input data points, the model may have gender-biased results.

[0075] Data 220 for bias(A) is sent to the bias(A) evaluator 230. The bias(A) evaluator 230 may be any system, evaluator, score / result generator, or other entity configured to process predictive data and attempt to detect bias for the model. Examples of bias evaluators include the Generalized Entropy Index (https: / / doi.org / 10.1145 / 3219819.3220046), Differential Fairness and Bias Amplification (DFBA) (https: / / arxiv.org / pdf / 1807.08362), Wino & gender bias score (http: / / web.cs.ucla.edu / ~kwchang / bibliography / zhao2018gender), LOGAN (http: / / web.cs.ucla.edu / ~kwchang / bibliography / zhao2020logan), etc.

[0076] The bias(A) evaluator 230 may use any number of techniques to determine bias results for the prediction data. In some embodiments, the bias(A) evaluator 230 compares each prediction result value with the corresponding synthetic input data point to determine whether the prediction result deviates from the expected prediction result. For example, the synthetic dataset may be used to determine the expected statistical range of values ​​for the prediction result values. The prediction result values ​​may then be compared with the statistical range to determine the likelihood that the prediction values ​​were generated according to a certain bias.

[0077] In addition to the predictive data 220 for bias (A), the bias (A) evaluator 220 may be configured to process the training data 240 for the trained model. Since the trained model 210 being evaluated is trained using the training data 240 for the trained model, the training data may also contain biases specific to the trained model. For example, the bias (A) evaluator 230 may be configured to extract one or more statistical metrics associated with the training dataset and determine whether the training dataset corresponds to biases associated with bias (A).

[0078] The bias(A) evaluator 230 generates and outputs an outcome 250 for bias(A) by utilizing bias evaluations performed on the predictive data 220 for bias(A) and the training data 240 for the trained model. In various embodiments, the outcome 250 for bias(A) is a bias result that may be included in a set of outcome data 162(A)~(N). The outcome 250 for bias(A) may correspond to one or more metrics related to the level of bias(A) detected for the trained ML model. For example, the outcome 250 for bias(A) may include a bias value / score that describes the relative level of bias detected for the trained model, evaluated based on the predictive data 220 for bias(A) and, in some embodiments, the training data 240 for the trained model.

[0079] Figure 3 shows a simplified diagram of the attribute classifier component of a bias evaluator as a service system in various embodiments. As shown in Figure 3, the shown attribute classifier includes several components through which data is generated. In particular, Figure 3 shows an attribute classifier 140 configured to take in a trained model and training dataset to generate one or more attributes.

[0080] As shown in Figure 3, the controller subsystem 130 may send the trained model 300 and / or the training dataset 310 to the attribute classifier 140. The trained model 300 is a trained ML model for which an evaluation for bias is required, and the training dataset 310 is the dataset on which the trained model was trained.

[0081] The trained model 300 may be received by the trained model scanner 320 of the attribute discriminator 140. The trained model scanner 320 may be a subsystem of the attribute discriminator 140 configured to parse the trained model 300 in order to detect one or more attributes of the trained model 300. For example, the trained model scanner 320 may be configured to receive, extract, or otherwise acquire model metadata from the trained model 300 in order to determine one or more attributes of input data points that may be input to the trained model 300.

[0082] The training dataset 310 may be received by the training data scanner 330 of the attribute classifier 140. The training data scanner 330 may be a subsystem of the attribute classifier 140 configured to parse the training dataset 310 to detect one or more attributes of the trained model 300. For example, the training data scanner 330 may be configured to parse the training dataset 310 to extract one or more column attributes of the training dataset 310 used to train the trained model 300.

[0083] The results output by the trained model scanner 320 and the training data scanner 330 may be integrated by the attribute aggregator 340 of the attribute classifier 140. The attribute aggregator 340 may be a subsystem of the attribute classifier 140 configured to edit and combine one or more attributes received from the trained model scanner 320 and the training data scanner 330 to form a combined set of one or more attributes. The attribute aggregator 340 may then output the integrated attributes 141 to the synthetic data generator 150.

[0084] Referring to the exemplary embodiment described above, the attribute classifier 140 may receive a trained model 300 that has been trained to take in input data points related to employee attributes and output predicted salary values ​​for employees. The attribute classifier 140 may send the trained model to a trained model scanner 320, which extracts metadata from the trained model 300 corresponding to the attributes input to the model. For example, based on the metadata, the trained model scanner may determine attributes such as the employee's name, gender, and job title. These attributes are edited and sent to a synthetic data generator 150 to generate synthetic data having the same attributes.

[0085] Figure 4 is a simplified flowchart illustrating an exemplary process for generating bias evaluation results for a trained machine learning model using a bias evaluator as a service system, according to various embodiments. The process shown in Figure 4 may be performed using hardware, or a combination thereof, in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods shown in and described below are intended to be illustrative and non-limiting. Figure 4 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, the processing may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as the embodiment shown in Figure 1, the process shown in Figure 4 may be performed by BEaaSS100.

[0086] Process 400 may be initiated in 402, where BEaaSS 100 receives a request to evaluate an ML model for bias, the request identifies the trained ML model to be evaluated, and the request selectively includes a training dataset used to train the ML model. Models may be received for any source from which a trained ML model for evaluation can be sent, including, for example, a user device 110 and / or a model catalog 120. The trained model data includes a trained ML model to be evaluated for bias. In some embodiments, the request identifies the location of the trained ML model, such as a URL from which the model can be retrieved. In some embodiments, the training dataset used to train the ML model is received as part of the request, and the training dataset may be further utilized to generate synthetic data for performing one or more bias checks on the trained ML model.

[0087] In 404, a set of model attributes for the trained model is determined. The set of model attributes may be determined by a subsystem of BEaaSS100, such as an attribute discriminator 140. The determination of the set of model attributes may be performed, for example, by extracting metadata from the trained model received in 402. One or more attributes correspond to one or more categories of inputs accepted by the ML model trained for evaluation. In various embodiments, determining one or more model attributes for the trained model includes evaluating and / or introspecting the trained ML model. For example, metadata from the trained ML model may be identified and parsed to determine the set of model attributes. In some embodiments, in 402, the training dataset is analyzed to determine the set of model attributes for the trained model.

[0088] Blocks 406-410 in Figure 4 arise for each bias check performed in the trained model. In 406, a synthetic dataset used for bias checks performed for the trained model is generated based on the set of model attributes determined in 404. The model attributes determined in 404 may be sent to a subsystem of BEaaSS100, such as a synthetic data generator 150, to generate the synthetic dataset. The synthetic dataset may be generated according to one or more methods for generating a synthetic dataset, such as by using a GAN ML model. In various embodiments, the synthetic dataset is generated based on one or more bias types for which bias evaluation is performed.

[0089] In 408, the synthetic data generated in 406 is input to the trained model received in 402 to generate prediction data for the synthetic dataset. For example, a bias checker, such as the bias checker shown in Figure 2, may be used to input the synthetic data into the trained ML model to generate prediction data. The generated set of predictions may be output as prediction data corresponding to one or more values ​​generated by inputting one or more synthetic input data points from the synthetic dataset generated in 406 into the trained ML model.

[0090] In 410, bias results are generated based on the prediction data generated in 408. The bias results may be evaluated by a bias evaluator, such as the bias evaluator shown in Figure 2. The bias results may be a value / score for a particular bias processed and output by a particular bias checker, or an integrated set of bias results processed and output by multiple bias checkers. In various embodiments, the bias results are generated by processing a set of predictions to be generated in 408 and determining one or more statistical distributions of the prediction data based on the synthetic dataset generated in 406. The bias results may therefore be based on the predicted values ​​or relative deviations of a set of predictions compared to a statistical distribution generated based on the synthetic dataset, in order to represent the predicted level of bias inherent in the predictions made by the trained ML model.

[0091] In some embodiments, quantifications such as bias scores are calculated based on the bias results generated in 410. For example, the score may be calculated based on the number of input data points in the input synthetic dataset, the expected number of predicted data points in the predicted data, and the actual predicted data points in the predicted data. For example, 1000 synthetic data points may be generated, including a field for gender. The synthetic data may include 500 data points including the "male" gender and 500 data points including the "female" gender. The synthetic data is then input into an ML model to generate predicted data. For example, the predicted data may represent a Boolean value of "true" or "false," and the synthetic data may be generated such that the mean of the synthetic dataset includes only the difference in the "male" or "female" data field. The expected output predicted dataset may be balanced such that 250 of each of the "male" and "female" data points are expected to correspond to the "true" output and 250 of each of the "male" and "female" data points are expected to correspond to the "false" output. However, the actual output prediction data may show a significantly larger proportion of "male" data points being "true" compared to "female" data points. This indicates gender bias in the ML model. The resulting score may be, for example, the ratio of the number of "female" data points predicted to be "true" to the number of "male" data points predicted to be "true". For example, if 430 "male" data points are predicted to be "true" and 31 "female" data points are predicted to be "true", the bias score may be expressed by the formula (1-(FT / MT))·100=92.8. After bias results have been generated for each bias check performed, the process proceeds to 412.

[0092] In 412, the output bias assessment report is generated based on the bias results generated in 410 for each bias check. The bias assessment report may include, for example, the bias results for each specific bias check, the bias score generated based on the bias results, a description of the level of bias for the trained ML model, and a visual representation of the level of bias detected. The report may be generated by, for example, a report generator 170 configured to generate a bias report for entity review. In another example, the results may be sent to an action subsystem 180 to determine or result in one or more actions that can be taken based on the generated bias results.

[0093] In 414, the bias assessment report generated in 412 is output. The bias assessment report may be output to entities such as the entity that made the request in 402. For example, the report may be generated and output to entities such as user device 110 or model catalog 120.

[0094] Figure 5 shows exemplary bias assessment reports generated by bias assessment as a service system in various embodiments. As shown in Figure 5, the report 500 may be provided in a readable format to entities such as customers or data scientists.

[0095] As shown in Figure 5, the bias assessment report may be provided in a human-readable format. The specific bias assessment report shown in Figure 5 may correspond to the exemplary embodiment described above relating to a bias detection process for a trained ML model to predict a salary value given input data points related to a certain employee characteristic. For example, report 500 may show the results of at least four different bias checks performed by four different bias checkers 161 for bias types “gender”, “race”, “religion”, and “language”. As shown in Figure 5, these bias types are scored as “WINO”, “DFBA”,<Custom_Religion> "<Custom_Language> Each model is scored using a bias checker. The models being evaluated are presented as "Employee Employment Models."

[0096] As shown in Figure 5, the results of the bias check for “gender” bias are displayed along with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. The indicator is shown as a visual indicator corresponding to the various levels of bias determined to be present in the trained ML model. As shown in Figure 5, for bias type: gender, the results of the bias check for gender detect a bias level above “intermediate,” but does not reach “high” for gender bias with respect to the trained ML model being evaluated. This may correspond to the determination that the trained ML model being evaluated tends to predict salary values ​​in a biased form based on the gender specified by the input data points, but gender does not affect the predicted salary. As shown in Figure 5, the bias has a score of “74” out of “100,” which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0097] As used herein, “score” may be a relative numerical representation of the level of bias detection in a trained ML model, as quantified by a bias evaluation system. For example, a score of 74 out of 100 may mean that 74 out of 100 results in the predicted data generated by processing an input synthetic dataset contained a determined form of bias. In other embodiments, the score may represent a relative statistical score of the model’s bias evaluation based on sampling of similar models. The bar graph shown adjacent to the score field in Figure 5 may show the relative level of bias as measured on a scale of bias detected in the model. For example, the representation of the gender bias level in Figure 5, which is above “medium” but not “high,” may indicate that the gender bias detected in the model was above the average level of bias in similar models, but not significantly higher by comparison.

[0098] Furthermore, as shown in Figure 5, the results of the bias check for “race” bias are displayed along with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. The indicator is a visual indicator corresponding to the various levels of bias determined to be present in the trained ML model. As shown in Figure 5, for bias type: race, the result of the bias check for race, i.e., the bias level, extends beyond “intermediate” and just short of “high.” This may correspond to the determination that the trained ML model being evaluated tends to predict salary values ​​in a biased form based on the race specified by the input data points, but race does not affect the predicted salary. As shown in Figure 5, the bias has a score of “94” out of “100,” which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0099] As shown in Figure 5, the results of the bias check for “religion” bias are displayed along with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. The indicator is a visual indicator corresponding to the various levels of bias determined to be present in the trained ML model. As shown in Figure 5, for bias type: religion, the result of the bias check for religion is detected as a bias level above “intermediate,” but does not reach “high” for the religion bias regarding the trained ML model being evaluated. This may correspond to the determination that the trained model being evaluated tends to predict salary values ​​in a biased form based on the religion specified by the input data points, but religion does not influence the predicted salary. As shown in Figure 5, the bias has a score of “51” out of “100,” which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0100] As shown in Figure 5, the results of the bias check for “language” bias are displayed along with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. The indicators are visual indicators corresponding to the various levels of bias determined to be present in the trained ML model. As shown in Figure 5, for bias type: language, the results of the bias check for religion include relatively low bias levels below “intermediate,” but language bias has been detected with respect to the trained ML model being evaluated. This may correspond to the determination that the trained ML model being evaluated tends to predict salary values ​​in a typically biased form based on the spoken language specified by the input data points, and that the spoken language does not influence the predicted salary. As shown in Figure 5, the bias has a score of “13” out of “100,” which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0101] Report 500 also includes an overall bias evaluation visual indicator and an overall combined score. The overall bias evaluation score may be an overall score generated by a component such as the overall score generator 171. In some embodiments, the overall score is generated based on the individual scores for the bias types described above. For example, a combined score of "58" out of "100" may be the overall average of the bias scores for the individual bias result scores. The indicator resides in a visual indicator corresponding to the overall average level of bias detected in the trained ML model. As shown in Figure 5, the result of the overall bias check for overall bias is a bias level that is above "intermediate" but does not reach "high". In various embodiments, the overall bias evaluation score may be generated as the average score of the other bias scores displayed in Report 500. Any mathematical or statistical combination of subscores for each bias may be used to form an overall score for display in the report. The corresponding bar graph of the overall bias score may show the relative level of overall bias for the model consisting of each bias type included in the report.

[0102] Report 500 also includes a status indicator of the action that should be taken with respect to the trained ML model, based on the generated overall score. For example, as shown in Figure 5, the field indicates that the model status of the trained ML model is “Bias Evaluation Failed,” indicating that the evaluated model (“Employee Hiring Model”) contains an unacceptable level of bias. “Failed” may indicate that the model should not be accepted by a catalog entity such as the model catalog. This may correspond to the decided action of rejecting the trained ML model from the model catalog 120 by determining at least “intermediate” overall bias for the trained ML model.

[0103] Exemplary Infrastructure-as-a-Service Implementation Figure 6 shows a bias assessment system for determining and reporting bias in trained ML models used to generate predictions, according to various embodiments. As described above, Infrastructure-as-a-Service (IaaS) is one specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider may supply various services associated with these infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering). Since these services may be policy-driven, IaaS users may enforce policies to drive load balancing to maintain application availability and performance.

[0104] In some cases, IaaS customers may access resources and services over a wide area network (WAN), such as the internet, and utilize the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workload and backups, and then install enterprise software on those VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0105] In most cases, the cloud computing model requires the participation of a cloud provider. A cloud provider may, but does not have to, be a third-party service specializing in providing IaaS (e.g., offering, renting, or selling). An entity may choose to deploy a private cloud, becoming its own provider of infrastructure services.

[0106] In some cases, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. This may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, the customer may be responsible for handling (OS), middleware and / or application deployment (e.g., self-service virtual machines, which can be spun up on demand), etc.

[0107] In some cases, IaaS provisioning may mean acquiring computers or virtual hosts for use, and then installing the necessary libraries or services on them. In most cases, a deployment does not include provisioning, and provisioning may be required to be performed first.

[0108] In some cases, there are two distinct issues with IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before everything is operational. Second, once everything is provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be solved by allowing the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how each of them works together) can be described declaratively. In some examples, once the topology is defined, it is possible to generate workflows that produce and / or manage the different components described in the configuration files.

[0109] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., a potentially on-demand pool of configurable and / or shared computing resources). In some examples, there may also be one or more security group rules and one or more virtual machines (VMs) provisioned to define how network security is set up. Other infrastructure elements such as load balancers, databases, or similar may also be provisioned. The infrastructure may evolve gradually as more infrastructure elements are desired and / or added.

[0110] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. In addition, the techniques described may enable infrastructure management within these environments. In some cases, a service team may write code that is to be deployed to one or more, but often many, different production environments (e.g., across various different geographical locations, sometimes worldwide). However, in some cases, the infrastructure to which the code is deployed must first be set up. In some cases, provisioning tools that allow provisioning to be done manually may be used to provision resources and / or, once the infrastructure is provisioned, deployment tools may be used to deploy the code.

[0111] Figure 6 is a block diagram 600 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 602 may be communicably coupled to a secure host tenancy 604 which may include a virtual cloud network (VCN) 606 and a secure host subnet 608. In some examples, the service operator 602 may use one or more client computing devices, which may be handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS and similar, and with the Internet, email, short message service (SMS), Blackberry®, or other communication protocols enabled. Alternatively, a client computing device could be a general-purpose personal computer, including, for example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. A client computing device could also be a workstation computer running any of various commercially available UNIX® or UNIX-like operating systems, including, without limitation, various GNU / Linux operating systems such as Google Chrome OS.Alternatively, or in addition, the client computing device may be any other electronic device, such as a thin client computer, an internet-connected gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device) and / or a VCN606 and / or a personal messaging device that can communicate over a network with internet access.

[0112] VCN606 may include a local peering gateway (LPG) 610 that can be communicatively coupled to Secure Shell (SSH) VCN612 via LPG610 contained within SSH VCN612. SSH VCN612 may include an SSH subnet 614, and SSH VCN612 may be communicatively coupled to control plane VCN616 via LPG610 contained within control plane VCN616. Furthermore, SSH VCN612 may be communicatively coupled to data plane VCN618 via LPG610. Control plane VCN616 and data plane VCN618 may be contained within a service tenancy 619 that may be owned and / or operated by an IaaS provider.

[0113] The control plane VCN616 may include a control plane demilitarized zone (DMZ) layer 620 that functions as a perimeter network (e.g., the portion of the corporate network between the corporate intranet and the external network). The DMZ-based servers may have limited liability and may help keep a security breach contained. In addition, the DMZ layer 620 may include one or more load balancer (LB) subnets 622, a control plane application layer 624 which may include application subnets 626, and a control plane data layer 628 which may include database (DB) subnets 630 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 622 included in the control plane DMZ layer 620 may be communicatively coupled to the application subnet 626 included in the control plane application layer 624 and an internet gateway 634 which may be included in the control plane VCN616, and the application subnet 626 may be communicatively coupled to the DB subnet 630 included in the control plane data layer 628, a service gateway 636, and a network address translation (NAT) gateway 638. The control plane VCN616 may include a service gateway 636 and a NAT gateway 638.

[0114] The control plane VCN616 may include a data plane mirror application layer 640, which may include an application subnet 626. The application subnet 626 included in the data plane mirror application layer 640 may include a virtual network interface controller (VNIC) 642 capable of running compute instance 644. Compute instance 644 can communicatively connect the application subnet 626 of the data plane mirror application layer 640 to an application subnet 626 that may be included in the data plane application layer 646.

[0115] The data plane VCN618 may include a data plane application layer 646, a data plane DMZ layer 648, and a data plane data layer 650. The data plane DMZ layer 648 may include an LB subnet 622 that can be communicatively coupled to the application subnet 626 of the data plane application layer 646 and the internet gateway 634 of the data plane VCN618. The application subnet 626 may be communicatively coupled to the service gateway 636 and the NAT gateway 638 of the data plane VCN618. The data plane data layer 650 may also include a DB subnet 630 that can be communicatively coupled to the application subnet 626 of the data plane application layer 646.

[0116] The Internet gateway 634 of the control plane VCN616 and data plane VCN618 may be communicatively coupled to a metadata management service 652 which may be communicatively coupled to the public internet 654. The public internet 654 may be communicatively coupled to the NAT gateway 638 of the control plane VCN616 and data plane VCN618. The service gateway 636 of the control plane VCN616 and data plane VCN618 may be communicatively coupled to a cloud service 656.

[0117] In some examples, the service gateway 636 of the control plane VCN616 and data plane VCN618 can make application programming interface (API) calls to the cloud service 656 without traversing the public internet 654. API calls from the service gateway 636 to the cloud service 656 can be one-way. The service gateway 636 can make an API call to the cloud service 656, and the cloud service 656 can send the requested data to the service gateway 636. However, the cloud service 656 may not initiate an API call to the service gateway 636.

[0118] In some examples, a secure host tenancy 604 may be directly connected to a service tenancy 619, which would otherwise be isolated. A secure host subnet 608 may communicate with an SSH subnet 614 via an LPG 610, which could enable bidirectional communication on a system that would otherwise be isolated. Connecting the secure host subnet 608 to the SSH subnet 614 may give the secure host subnet 608 access to other entities within the service tenancy 619.

[0119] The control plane VCN616 may allow users of service tenancy 619 to set up or otherwise provision desired resources. Desired resources provisioned in the control plane VCN616 may be deployed or otherwise used in the data plane VCN618. In some examples, the control plane VCN616 may be isolated from the data plane VCN618, and the data plane mirror application layer 640 of the control plane VCN616 may communicate with the data plane application layer 646 of the data plane VCN618 via VNIC642, which may be included in the data plane mirror application layer 640 and the data plane application layer 646.

[0120] In some examples, a system user or customer may make requests, such as create, read, update, or delete (CRUD) operations, through the public internet 654, which can communicate requests to the metadata management service 652. The metadata management service 652 can communicate requests to the control plane VCN 616 through the internet gateway 634. Requests may be received by the LB subnet 622, which is included in the control plane DMZ layer 620. The LB subnet 622 may determine that the request is valid, and in response to this determination, the LB subnet 622 may send the request to the application subnet 626, which is included in the control plane application layer 624. If the request is authenticated and requests a call to the public internet 654, the call to the public internet 654 may be sent to the NAT gateway 638, which is capable of making calls to the public internet 654. Memory that may be desired to be stored by the request may be stored in the DB subnet 630.

[0121] In some cases, the data plane mirror application layer 640 can facilitate direct communication between the control plane VCN616 and the data plane VCN618. For example, it may be desirable that changes, updates, or otherwise appropriate modifications to settings be applied to resources contained in the data plane VCN618. Through VNIC642, the control plane VCN616 can communicate directly with the resources contained in the data plane VCN618, thereby enabling changes, updates, or otherwise appropriate modifications to the settings of these resources.

[0122] In some embodiments, the control plane VCN616 and data plane VCN618 may be included in the service tenancy 619. In this case, the system user or customer does not have to own or operate either the control plane VCN616 or the data plane VCN618. The IaaS provider may own or operate the control plane VCN616 and data plane VCN618, and both may be included in the service tenancy 619. This embodiment can enable network isolation, which may prevent a user or customer from interacting with resources of other users or other customers. This embodiment may also allow the system user or customer to store databases privately without having to rely on the public internet 654, which may not have the desired level of security for storage.

[0123] In other embodiments, the LB subnet 622 included in the control plane VCN616 may be configured to receive signals from the service gateway 636. In this embodiment, the control plane VCN616 and the data plane VCN618 may be configured to be called by the IaaS provider's customer without calling the public internet 654. The IaaS provider's customer may prefer this embodiment because the database used by the customer may be stored in a service tenancy 619 that may be controlled by the IaaS provider and isolated from the public internet 654.

[0124] Figure 7 is a block diagram 700 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 702 (e.g., service operator 602 in Figure 6) may be communicatively coupled to a secure host tenancy 704 (e.g., secure host tenancy 604 in Figure 6), which may include a virtual cloud network 706 (e.g., VCN606 in Figure 6) and a secure host subnet 708 (e.g., secure host subnet 608 in Figure 6). VCN706 may include a local peering gateway (LPG) 710 (e.g., LPG610 in Figure 6), which may be communicatively coupled to a secure shell (SSH) VCN712 (e.g., SSH VCN612 in Figure 6) via an LPG610 included in an SSH VCN712. SSH VCN712 may include SSH subnet 714 (e.g., SSH subnet 614 in Figure 6), and SSH VCN712 may be communicably coupled to control plane VCN716 (e.g., control plane VCN616 in Figure 6) via LPG710 included in control plane VCN716. Control plane VCN716 may be included in service tenancy 719 (e.g., service tenancy 619 in Figure 6), and data plane VCN718 (e.g., data plane VCN618 in Figure 6) may be included in customer tenancy 721, which may be owned or operated by a user or customer of the system.

[0125] The control plane VCN716 may include a control plane DMZ layer 720 (e.g., control plane DMZ layer 620 in Figure 6) which may include an LB subnet 722 (e.g., LB subnet 622 in Figure 6), a control plane application layer 724 (e.g., control plane application layer 624 in Figure 6) which may include an application subnet 726 (e.g., application subnet 626 in Figure 6), and a control plane data layer 728 (e.g., control plane data layer 628 in Figure 6) which may include a database (DB) subnet 730 (e.g., similar to DB subnet 630 in Figure 6). The LB subnet 722 included in the control plane DMZ layer 720 may be communicatively coupled to the application subnet 726 included in the control plane application layer 724 and to an internet gateway 734 (e.g., internet gateway 634 in Figure 6) which may be included in the control plane VCN 716. The application subnet 726 may be communicatively coupled to the DB subnet 730 included in the control plane data layer 728, to a service gateway 736 (e.g., service gateway in Figure 6) and to a network address translation (NAT) gateway 738 (e.g., NAT gateway 638 in Figure 6). The control plane VCN 716 may include the service gateway 736 and the NAT gateway 738.

[0126] The control plane VCN 716 may include a data plane mirror application layer 740 (e.g., the data plane mirror application layer 640 in Figure 6) which may include an application subnet 726. The application subnet 726 included in the data plane mirror application layer 740 may include a virtual network interface controller (VNIC) 742 (e.g., VNIC 642) capable of running a compute instance 744 (e.g., similar to compute instance 644 in Figure 6). The compute instance 744 can facilitate communication between the application subnet 726 of the data plane mirror application layer 740 and the application subnet 726 included in the data plane application layer 746 (e.g., the data plane application layer 646 in Figure 6) via the VNIC 742 included in the data plane mirror application layer 740 and the VNIC 742 included in the data plane application layer 746.

[0127] The Internet gateway 734 included in the control plane VCN 716 can be communicably coupled to a metadata management service 752 (e.g., metadata management service 652 in Figure 6), which can be communicably coupled to the public internet 754 (e.g., public internet 654 in Figure 6). The public internet 754 can be communicably coupled to a NAT gateway 738 included in the control plane VCN 716. The service gateway 736 included in the control plane VCN 716 can be communicably coupled to a cloud service 756 (e.g., cloud service 656 in Figure 6).

[0128] In some examples, the data plane VCN718 may be included in a customer tenancy 721. In this case, the IaaS provider may provide a control plane VCN716 for each customer, and the IaaS provider may set up a unique compute instance 744 included in a service tenancy 719 for each customer. Each compute instance 744 may enable communication between the control plane VCN716 included in the service tenancy 719 and the data plane VCN718 included in the customer tenancy 721. The compute instance 744 may enable resources provisioned in the control plane VCN716 included in the service tenancy 719 to be deployed or otherwise used in the data plane VCN718 included in the customer tenancy 721.

[0129] In another example, an IaaS provider's customer may have a database residing in customer tenancy 721. In this example, the control plane VCN 716 may include a data plane mirror app layer 740, which may include an app subnet 726. The data plane mirror app layer 740 may reside in data plane VCN 718, but does not have to reside in data plane VCN 718. That is, the data plane mirror app layer 740 may have access to customer tenancy 721, but does not have to reside in data plane VCN 718 or be owned or operated by the IaaS provider's customer. The data plane mirror app layer 740 may be configured to make calls to data plane VCN 718, but does not have to be configured to make calls to any entity included in control plane VCN 716. Customers may want to deploy or otherwise use resources in the data plane VCN718 that were provisioned in the control plane VCN716, and the data plane mirror application layer 740 can facilitate the customer's desired deployment or otherwise use of resources.

[0130] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN718. In this embodiment, the customer can determine what the data plane VCN718 can access, and the customer may restrict access from the data plane VCN718 to the public internet 754. The IaaS provider may not be able to apply filters or otherwise control the data plane VCN718's access to any external network or database. The application of filters and controls by the customer to the data plane VCN718 included in the customer tenancy 721 can help isolate the data plane VCN718 from other customers and the public internet 754.

[0131] In some embodiments, a cloud service 756 may be called by a service gateway 736 to access services that may not exist on the public internet 754, control plane VCN 716, or data plane VCN 718. The connection between the cloud service 756 and the control plane VCN 716 or data plane VCN 718 may not be live or continuous. The cloud service 756 may reside on different networks owned or operated by the IaaS provider. The cloud service 756 may be configured to receive calls from the service gateway 736 and not to receive calls from the public internet 754. Some cloud services 756 may be isolated from other cloud services 756, and the control plane VCN 716 may be isolated from cloud services 756 that do not need to be in the same region as the control plane VCN 716. For example, the control plane VCN 716 may be located in "Region 1", and the cloud service "Deployment 8" may be located in Region 1 and "Region 2". When a call to deployment 8 is made by a service gateway 736 included in a control plane VCN716 located in region 1, the call may be sent to deployment 8 in region 1. In this example, the control plane VCN716 or deployment 8 in region 1 does not need to be communicatively coupled to or otherwise communicate with deployment 8 in region 2.

[0132] Figure 8 is a block diagram 800 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 802 (e.g., service operator 602 in Figure 6) may be communicatively coupled to a secure host tenancy 804 (e.g., secure host tenancy 604 in Figure 6), which may include a virtual cloud network (VCN) 806 (e.g., VCN606 in Figure 6) and a secure host subnet 808 (e.g., secure host subnet 608 in Figure 6). VCN806 may include an LPG810 (e.g., LPG610 in Figure 6), which may be communicatively coupled to an SSH VCN812 (e.g., SSH VCN612 in Figure 6) via an LPG810 contained in an SSH VCN812. SSH VCN812 may include SSH subnet 814 (e.g., SSH subnet 614 in Figure 6), and SSH VCN812 may be communicatively coupled to control plane VCN816 (e.g., control plane VCN616 in Figure 6) via LPG810 included in control plane VCN816, and may be communicatively coupled to data plane VCN818 (e.g., data plane 618 in Figure 6) via LPG810 included in data plane VCN818. Control plane VCN816 and data plane VCN818 may be included in service tenancy 819 (e.g., service tenancy 619 in Figure 6).

[0133] The control plane VCN816 may include a control plane DMZ layer 820 (e.g., control plane DMZ layer 620 in Figure 6) which may include a load balancer (LB) subnet 822 (e.g., LB subnet 622 in Figure 6), a control plane application layer 824 (e.g., control plane application layer 624 in Figure 6) which may include an application subnet 826 (similar to application subnet 626 in Figure 6), and a control plane data layer 828 (control plane data layer 628 in Figure 6) which may include a DB subnet 830. The LB subnet 822 included in the control plane DMZ layer 820 may be communicatively coupled to the application subnet 826 included in the control plane application layer 824 and to an internet gateway 834 (e.g., internet gateway 634 in Figure 6) which may be included in the control plane VCN 816. The application subnet 826 may be communicatively coupled to the DB subnet 830 included in the control plane data layer 828, to a service gateway 836 (e.g., service gateway in Figure 6) and to a network address translation (NAT) gateway 838 (e.g., NAT gateway 638 in Figure 6). The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0134] The data plane VCN818 may include a data plane application layer 846 (e.g., data plane application layer 646 in Figure 6), a data plane DMZ layer 848 (data plane DMZ layer 648 in Figure 6), and a data plane data layer 850 (e.g., data plane data layer 650 in Figure 6). The data plane DMZ layer 848 may include an LB subnet 822 that can be communicatively coupled to the trusted application subnet 860 and untrusted application subnet 862 of the data plane application layer 846, and to the internet gateway 834 included in the data plane VCN818. The trusted application subnet 860 may be communicatively coupled to the service gateway 836 included in the data plane VCN818, the NAT gateway 838 included in the data plane VCN818, and the DB subnet 830 included in the data plane data layer 850. An untrusted application subnet 862 may be communicatively coupled to a service gateway 836 included in the data plane VCN 818 and to a DB subnet 830 included in the data plane data layer 850. The data plane data layer 850 may include a DB subnet 830 that can be communicatively coupled to a service gateway 836 included in the data plane VCN 818.

[0135] An untrusted application subnet 862 may contain one or more primary subnets that can be communicatively coupled to tenant virtual machines (VMs) 866(1)-(N). Each tenant VM 866(1)-(N) may be communicatively coupled to each application subnet 867(1)-(N) that may be included in each container egress VCN 868(1)-(N) that may be included in each customer tenancy 870(1)-(N). Each secondary VNIC 872(1)-(N) can facilitate communication between the untrusted application subnet 862 included in the data plane VCN 818 and the application subnets included in the container egress VCN 868(1)-(N). Each container egress VCN 868(1)-(N) may include a NAT gateway 838 that can be communicatively coupled to the public internet 854 (e.g., public internet 654 in Figure 6).

[0136] The Internet gateway 834 included in the control plane VCN816 and data plane VCN818 can be communicatively coupled to a metadata management service 852 (e.g., metadata management system 652 in Figure 6), which can be communicatively coupled to the public internet 854. The public internet 854 can be communicatively coupled to a NAT gateway 838 included in the control plane VCN816 and data plane VCN818. The service gateway 836 included in the control plane VCN816 and data plane VCN818 can be communicatively coupled to a cloud service 856.

[0137] In some embodiments, the data plane VCN818 may be integrated with the customer tenancy 870. This integration may be useful or desirable for the IaaS provider's customer in several cases, such as when support may be desired when executing code. The customer may provide code to be executed that may be disruptive, communicate with other customer resources, or otherwise produce undesirable effects. In response, the IaaS provider may decide whether to execute the code provided to the IaaS provider by the customer.

[0138] In some examples, an IaaS provider's customer may grant temporary network access to the IaaS provider and request a function to be attached to a data plane tier application 846. The code for executing the function may be executed in VM866(1)~(N), and the code does not need to be configured to run anywhere else in the data plane VCN818. Each VM866(1)~(N) may be connected to a single customer tenancy 870. Each container 871(1)~(N) contained within VM866(1)~(N) may be configured to execute the code. In this case, dual isolation may exist (for example, containers 871(1)-(N) execute code, and containers 871(1)-(N) may be contained in at least VM866(1)-(N) contained in the untrusted app subnet 862), which may help prevent erroneous or otherwise undesirable code from damaging the IaaS provider's network or the network of a different customer. Containers 871(1)-(N) may be communicatively coupled to customer tenancy 870 and may be configured to send and receive data with customer tenancy 870. Containers 871(1)-(N) do not have to be configured to send and receive data with any other entities in the data plane VCN818. Once the execution of the code is complete, the IaaS provider may kill or otherwise discard containers 871(1)-(N).

[0139] In some embodiments, a trusted application subnet 860 may execute code that may be owned or operated by the IaaS provider. The trusted application subnet 860 may be communicatively coupled to a DB subnet 830 and configured to perform CRUD operations in the DB subnet 830. An untrusted application subnet 862 may be communicatively coupled to a DB subnet 830, but in this embodiment, the untrusted application subnet may be configured to perform read operations in the DB subnet 830. Containers 871(1)~(N) that may be contained in each customer's VM866(1)~(N) and that may execute code from the customer do not have to be communicatively coupled to a DB subnet 830.

[0140] In other embodiments, the control plane VCN816 and data plane VCN818 do not need to be directly communicatively coupled. In this embodiment, direct communication between the control plane VCN816 and the data plane VCN818 is not required. However, communication can occur indirectly through at least one method. An LPG810 that facilitates communication between the control plane VCN816 and the data plane VCN818 may be installed by the IaaS provider. In another example, the control plane VCN816 or the data plane VCN818 can make a call to the cloud service 856 via the service gateway 836. For example, a call from the control plane VCN816 to the cloud service 856 may include a request for a service that can communicate with the data plane VCN818.

[0141] Figure 9 is a block diagram 900 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 902 (e.g., service operator 602 in Figure 6) may be communicatively coupled to a secure host tenancy 904 (secure host tenancy 604 in Figure 6), which may include a virtual cloud network (VCN) 906 (e.g., VCN606 in Figure 6) and a secure host subnet 908 (e.g., secure host subnet 608 in Figure 6). VCN906 may include an LPG910 (e.g., LPG610 in Figure 6), which may be communicatively coupled to an SSH VCN912 (e.g., SSH VCN612 in Figure 6) via an LPG910 contained in an SSH VCN912. SSH VCN912 may include SSH subnet 914 (e.g., SSH subnet 614 in Figure 6), and SSH VCN912 may be communicatively coupled to control plane VCN916 (e.g., control plane VCN616 in Figure 6) via LPG910 included in control plane VCN916, and may be communicatively coupled to data plane VCN918 (e.g., data plane 618 in Figure 6) via LPG910 included in data plane VCN918. Control plane VCN916 and data plane VCN918 may be included in service tenancy 919 (e.g., service tenancy 619 in Figure 6).

[0142] The control plane VCN916 may include a control plane DMZ layer 920 (e.g., control plane DMZ layer 620 in Figure 6) which can include an LB subnet 922 (e.g., LB subnet 622 in Figure 6), a control plane application layer 924 (e.g., control plane application layer 624 in Figure 6) which can include an application subnet 926 (e.g., application subnet 626 in Figure 6), and a control plane data layer 928 (e.g., control plane data layer 628 in Figure 6) which can include a DB subnet 930 (e.g., DB subnet 830 in Figure 8). The LB subnet 922 included in the control plane DMZ layer 920 may be communicatively coupled to the application subnet 926 included in the control plane application layer 924 and to an internet gateway 934 (e.g., internet gateway 634 in Figure 6) which may be included in the control plane VCN 916. The application subnet 926 may be communicatively coupled to the DB subnet 930 included in the control plane data layer 928, to a service gateway 936 (e.g., service gateway in Figure 6) and to a network address translation (NAT) gateway 938 (e.g., NAT gateway 638 in Figure 6). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0143] The data plane VCN918 may include a data plane application layer 946 (e.g., data plane application layer 646 in Figure 6), a data plane DMZ layer 948 (e.g., data plane DMZ layer 648 in Figure 6), and a data plane data layer 950 (e.g., data plane data layer 650 in Figure 6). The data plane DMZ layer 948 may include a trusted application subnet 960 (e.g., trusted application subnet 860 in Figure 8) and an untrusted application subnet 962 (e.g., untrusted application subnet 862 in Figure 8) of the data plane application layer 946, and an LB subnet 922 which can be communicatively coupled to the internet gateway 934 included in the data plane VCN918. The trusted application subnet 960 may be communicatively coupled to a service gateway 936 included in the data plane VCN918, a NAT gateway 938 included in the data plane VCN918, and a DB subnet 930 included in the data plane data layer 950. An untrusted application subnet 962 may be communicatively coupled to a service gateway 936 included in the data plane VCN 918 and to a DB subnet 930 included in the data plane data layer 950. The data plane data layer 950 may include a DB subnet 930 that can be communicatively coupled to a service gateway 936 included in the data plane VCN 918.

[0144] An untrusted application subnet 962 may include primary VNICs 964(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N) residing within the untrusted application subnet 962. Each tenant VM 966(1)-(N) can execute code in its respective container 967(1)-(N) and may be communicatively coupled to an application subnet 926 that may be included in a dataplane application layer 946 that may be included in a container egress VCN 968. Each secondary VNIC 972(1)-(N) can facilitate communication between the untrusted application subnet 962 included in the dataplane VCN 918 and the application subnet included in the container egress VCN 968. The container egress VCN may include a NAT gateway 938 that can be communicatively coupled to the public internet 954 (e.g., public internet 654 in Figure 6).

[0145] The Internet gateway 934 included in the control plane VCN916 and data plane VCN918 can be communicatively coupled to a metadata management service 952 (e.g., metadata management system 652 in Figure 6), which can be communicatively coupled to the public internet 954. The public internet 954 can be communicatively coupled to a NAT gateway 938 included in the control plane VCN916 and data plane VCN918. The service gateway 936 included in the control plane VCN916 and data plane VCN918 can be communicatively coupled to a cloud service 956.

[0146] In some examples, the pattern shown by the architecture in block diagram 900 of Figure 9 may be considered an exception to the pattern shown by the architecture in block diagram 800 of Figure 8, and may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., a disconnected region). Each container 967(1)~(N) contained within VM966(1)~(N) for each customer can be accessed by the customer in real time. Each container 967(1)~(N) may be configured to make calls to each secondary VNIC 972(1)~(N) contained within the application subnet 926 of the data plane application layer 946, which may be contained within the container egress VCN968. The secondary VNICs 972(1)~(N) can send calls to the NAT gateway 938, which may send calls to the public internet 954. In this example, containers 967(1)-(N), which may be accessed in real time by customers, may be isolated from the control plane VCN916 and from other entities included in the data plane VCN918. Containers 967(1)-(N) may also be isolated from resources from other customers.

[0147] In another example, a customer can use containers 967(1) to (N) to call cloud service 956. In this example, the customer may execute code in containers 967(1) to (N) to request a service from cloud service 956. Containers 967(1) to (N) can send this request to secondary VNICs 972(1) to (N), which can send the request to the NAT gateway, which can send the request to the public internet 954. The public internet 954 can send the request to LB subnet 922, which is included in control plane VCN 916, via internet gateway 934. In response to the decision that the request is valid, the LB subnet can send the request to application subnet 926, which can send the request to cloud service 956 via service gateway 936.

[0148] It should be acknowledged that the IaaS architectures 600, 700, 800, and 900 shown in the drawings may have other components besides those shown. Furthermore, the embodiments shown in the drawings are only some examples of cloud infrastructure systems that may incorporate certain embodiments. In some other embodiments, the IaaS system may have more or fewer components than those shown in the drawings, may combine two or more components, or may have different configurations or arrangements of components.

[0149] In some embodiments, the IaaS system described herein may include a suite of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI), provided by the Assignee.

[0150] Figure 10 shows an exemplary computer system 1000 that can be used to implement various embodiments. System 1000 may be used to implement any of the computer systems described above. As shown in the drawing, computer system 1000 includes a processing unit 1004 that communicates with a number of peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer-readable storage medium 1022 and system memory 1010.

[0151] The bus subsystem 1002 provides a mechanism for various components and subsystems of the computer system 1000 to communicate with each other as intended. Although the bus subsystem 1002 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1002 may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using one of various bus architectures. For example, such architectures may include industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, enhanced ISA (EISA) buses, video electronics standards association (VESA) local buses, and peripheral component interconnect (PCI) buses, which may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0152] The processing unit 1004, which can be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core or multi-core processors. In one embodiment, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, each containing a single or multi-core processor. In other embodiments, the processing unit 1004 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0153] In various embodiments, the processing unit 1004 can execute various programs in response to program code and can maintain a large number of programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the processor 1004 and / or the memory subsystem 1018. With appropriate programming, the processor 1004 can provide the various functions described above. The computer system 1000 may also include a processing acceleration unit 1006, which may include a digital signal processor (DSP), a dedicated processor, and / or similar.

[0154] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion detection and / or gesture recognition devices, such as Microsoft Kinect® motion sensor, which enable the user to control and interact with input devices, such as Microsoft Xbox® 360 game controllers, through a natural user interface using gestures and voice commands. User interface input devices may also include eye gesture recognition devices, such as Google Glass® blink detectors, which detect eye movements from the user (e.g., blinks while taking photos and / or making menu selections) and translate eye gestures as input to an input device (e.g., Google Glass®). In addition, the user interface input device may include a voice recognition detection device that enables the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0155] User interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. In addition, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and similar devices.

[0156] User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, and similar devices. Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or other computer. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphic, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0157] The computer system 1000 may include a storage subsystem 1018 containing software elements currently located in the system memory 1010. The system memory 1010 may store loadable and executable program instructions in the processing unit 1004 and data generated during the execution of these programs.

[0158] Depending on the configuration and type of the computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM generally contains data and / or program modules that are immediately accessible to the processing unit 1004 and / or currently being operated and executed by the processing unit 1004. In some implementations, the system memory 1010 may contain a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS) containing basic routines that help transmit information between elements within the computer system 1000, such as during startup, may generally be stored in ROM. For example, but not limited to, the system memory 1010 also contains application programs 1012, program data 1014, and an operating system 1016, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), etc. For example, Operating System 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, and similar ones), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 6 OS, and Palm® OS.

[0159] The storage subsystem 1018 may also provide a tangible computer-readable storage medium for storing basic programming and data structures that provide the functionality of several embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provides the functionality described above may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with this disclosure.

[0160] The memory subsystem 1018 may also include a computer-readable storage medium reader 1020 which can be further connected to the computer-readable storage medium 1022. Together, and selectively in combination with the system memory 1010, the computer-readable storage medium 1022 may comprehensively represent remote, local, fixed, and / or removable storage devices and storage media for storing, transmitting, and retrieving computer-readable information temporarily and / or more permanently.

[0161] The computer-readable storage medium 1022, including code or a portion of code, may also include any suitable medium known or used in the art, including, but not limited to, volatile and non-volatile, removable and non-removable media, storage media and communication media, implemented in any way or technique for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer-readable media. This may also include non-tangible computer-readable media such as data signals, data transmissions, or any other media that can be used to transmit desired information and can be accessed by the computing system 1000.

[0162] For example, the computer-readable storage medium 1022 may include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD-ROM, Blu-Ray® disc, or other optical medium. The computer-readable storage medium 1022 may also include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital videotapes, and similar devices. The computer-readable storage medium 1022 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, and similar devices, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media may provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 1000.

[0163] The communication subsystem 1024 provides interfaces to other computer systems and networks. It acts as an interface for receiving data from other systems and transmitting data from computer system 1000 to other systems. For example, the communication subsystem 1024 may enable computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1024 may include modern data network technologies (e.g., using cellular technology, 3G, 4G, or EDGE (Enhanced data rates for global evolution), WiFi (IEEE 802.11 family standards), or other mobile communication technologies, or any combination thereof), radio frequency (RF) transceiver components for accessing radio voice and / or data networks, global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communication subsystem 1024 may provide wired network connectivity (e.g., Ethernet®) in addition to or instead of the wireless interface.

[0164] In some embodiments, the communication subsystem 1024 may receive input communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates, and the like for one or more users who may use the computer system 1000.

[0165] For example, the communication subsystem 1024 may be configured to receive real-time data feeds 1026 from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, and web feeds such as rich site summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0166] In addition, the communication subsystem 1024 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1028 of real-time events and / or event updates 1030, which may be continuous or infinite and not inherently have a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and similar.

[0167] The communication subsystem 1024 may be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, and similar to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 1000.

[0168] Computer system 1000 can be one of various types, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing systems.

[0169] Due to the constantly changing nature of computers and networks, the description of the computer system 1000 shown in the drawings is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in the drawings. For example, customized hardware may also be used and / or certain elements may be implemented in hardware, firmware, software (including applets) or a combination thereof. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosures and teachings provided herein, those skilled in the art will recognize other means and / or methods for carrying out various embodiments.

[0170] While specific embodiments are described, the scope of the disclosure also includes various modifications, changes, alternative structures, and equivalents. The embodiments are not limited to operation within a particular data processing environment and are free to operate within multiple data processing environments. In addition, while the embodiments are described using specific sequences of transactions and steps, it should be apparent to those skilled in the art that the claims are not limited to the described sequences of transactions and steps. Various features and aspects of the embodiments described above may be used individually or in combination.

[0171] Furthermore, while embodiments are described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of the disclosed embodiments. Embodiments may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same or different processors in any combination. Thus, where a component or module is described as being configured to perform a certain operation, such configuration may be realized, for example, by designing electronic circuits for performing the operation, by programming programmable electronic circuits (such as a microprocessor) for performing the operation, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0172] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will become clear that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader intent and scope set forth in the claims. Thus, while specific embodiments are described, they are not intended to be limiting. Various modifications and equivalents fall within the scope of the embodiments described in the claims.

[0173] In the context describing the disclosed embodiments (the context of the claims set forth below), the use of the terms “a,” “an,” and “the” and similar reference subjects should be interpreted as including both singular and plural unless otherwise indicated herein or the context clearly contradicts it. The terms “comprising,” “having,” “including,” and “containing” should be interpreted as unrestricted terms (i.e., “including, but not limited to”) unless otherwise indicated herein. The term “connected” should be interpreted as including, attaching, or joining, partially or entirely, even if there is something intervening. Descriptions of ranges of values ​​herein are intended only as an abbreviated way of referring individually to each separate value included within that range unless otherwise indicated herein, and each separate value is incorporated into the specification as individually enumerated herein. All methods described herein may be performed in any appropriate order unless otherwise indicated herein or the context clearly contradicts it. All and every example or exemplary language provided herein (e.g., "etc.") is intended solely to further illustrate certain embodiments and does not limit the scope of the disclosed art. The language in the specification should not be construed as indicating any element not shown in the claims as essential for the implementation of the embodiments shown in the claims.

[0174] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as commonly used to indicate that an item, term, etc., may be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise specified. Therefore, disjunctive language is not generally intended, nor should it be, to imply that a particular embodiment requires the presence of at least one X, at least one Y, or at least one Z, respectively.

[0175] Preferred embodiments, including best modes known for carrying out various embodiments, are described herein. Variations of these preferred embodiments may become apparent to those skilled in the art by reading the above description. Those skilled in the art should be able to adopt such variations as appropriate, and the described embodiments may be carried out in ways other than those specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter described in the claims attached to this disclosure, as permitted by applicable law. Furthermore, all combinations of the elements described above in all possible variations thereof are encompassed herein unless otherwise indicated herein.

[0176] All references cited herein, including publications, patent applications, and patents, are incorporated herein by reference to the same extent as each reference is shown individually and specifically as being incorporated by reference, and as being shown as the entirety of this specification.

[0177] In this specification, novel embodiments are described with reference to specific embodiments, but those skilled in the art will recognize that this disclosure is not limited thereto. The various features and embodiments of the embodiments described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method by which a computer performs an action. For a trained model to be evaluated, the computing system includes determining a set of model attributes for the trained model. The aforementioned method, The computing system further includes generating a first synthetic dataset used for a first bias check performed for the trained model based on the set of model attributes, wherein the first bias check is configured to evaluate the trained model with respect to a first bias type, and the first synthetic dataset includes a plurality of data points. The aforementioned method, The process further includes using the trained model to generate first predictive data for the first synthetic dataset, the first predictive data comprising a plurality of first predictive values ​​generated by the trained model for the plurality of data points in the first synthetic dataset. The aforementioned method, The computing system further includes generating a first bias result for the first bias type based on the first prediction data, The aforementioned method, A method performed by a computer, further comprising the computing system generating a bias assessment report for the trained model, wherein the bias assessment report includes information indicating the first bias result.

2. The method performed by a computer according to claim 1, wherein the first bias result includes one or more bias values ​​generated based on the first prediction data.

3. The method performed by a computer according to claim 2, wherein the bias evaluation report includes the first bias result and the one or more bias values, and the method further includes outputting the bias evaluation report.

4. The aforementioned method, The process involves comparing at least one of the aforementioned bias values ​​with a bias threshold value, A method performed by a computer according to claim 2, further comprising deciding whether to accept or reject, based on the comparison, whether to include the trained model in the group of trained models.

5. The aforementioned method, The computing system further includes generating a second synthetic dataset used for a second bias check performed for the trained model based on the set of model attributes, wherein the second bias check is configured to evaluate the trained model with respect to a second bias type, and the second synthetic dataset includes a plurality of data points. The aforementioned method, Using the trained model, generate second prediction data for the second synthetic dataset. The method performed by a computer according to claim 1, further comprising the computing system generating a second bias result for the second bias type based on the second prediction data.

6. The aforementioned method, The computing system generates a bias score based on the first bias result and the second bias result, A method performed by a computer according to claim 5, further comprising deciding whether to accept or reject the trained model from the group of trained models based on the generated bias score.

7. A method performed by a computer according to claim 1, wherein determining the set of model attributes includes processing the trained model to determine at least one model attribute in the set of model attributes.

8. A computer-based method according to claim 1, wherein determining the set of model attributes includes determining at least one model attribute in the set of model attributes based on an analysis of the training data used to train and generate the trained model.

9. The aforementioned method, Determining the training data used to train and generate the aforementioned trained model, The method performed by a computer according to claim 1, further comprising the computing system generating a second bias result for the first bias type based on the training data, wherein generating the first bias result further comprises generating based on the generated second bias result.

10. The method performed by a computer according to claim 1, wherein generating the first synthetic dataset comprises the computing system generating the first synthetic dataset using a generative neural network machine learning model based on the set of model attributes for the trained model.

11. The aforementioned trained model is a neural network, The method according to claim 1, wherein the prediction data further includes at least one value generated by the output layer of the neural network.

12. One or more processors, A system comprising: a memory that, when executed by the one or more processors, includes instructions that cause the one or more processors to perform the method according to any one of claims 1 to 11.

13. A program for causing one or more processors to perform the method according to any one of claims 1 to 11.

Images