Techniques for assessing bias in trained models

Abstract

A system configured to perform various bias checks on a machine learning (ML) model to identify one or more biases, if any, that may be inherent to the ML model is disclosed. Bias assessment results generated by performing the checks are then reported to users, such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. The bias assessment system performs one or more bias checks by generating a synthetic dataset using attributes present in the ML model or training dataset used to train the ML model. Prediction data is then generated by inputting synthetically generated input data points of the synthetic dataset into the ML model. The prediction data is then processed and evaluated for bias. The results of the evaluation may be compiled into a bias assessment report.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Nonprovisional Patent Application No. 17 / 508,734, entitled “TECHNIQUES FOR TRAINED MODEL BIAS ASSESSMENT,” filed on October 22, 2021, the entire contents of which are incorporated herein by reference in their entirety for all purposes. [Background technology]

[0002] background Recent years have seen a rapid increase in the adoption of artificial intelligence (AI) and machine learning (ML) solutions in a variety of different industries and applications. For a typical ML solution, in the training phase, an ML model is trained and validated using certain training and validation datasets. Once the model reaches an acceptable level of accuracy in the training phase, the model is then deployed to a manufacturing environment where it is used to make predictions on real-time production data inputs. However, if the particular training dataset used to train the ML model contains biased data, the trained ML model will also make biased predictions.

[0003] Bias is a common, but not trivial, problem in ML that is responsible for many erroneous predictions made by ML models and can lead to model instability in real-world scenarios. For example, a biased ML model may not only fail to predict correct information, but may also predict incorrect information that the ML model represents as a good prediction. This creates problems for services that host trained ML models for their customers, and these problems are magnified when the service hosts ML models that are not trained under the service's supervision. Bias is not a problem that can be solved by simply increasing the size of the training dataset, and in fact, a larger biased training dataset may only intensify undesirable biased behavior in the trained ML model. Often, the trained ML model itself is a "black box" entity, and bias often cannot be detected by simply considering the organization of the trained ML model itself. Summary of the Invention [Means for solving the problem]

[0004] overview The present disclosure relates to assessing bias in trained machine learning (ML) models. A system configured to perform various bias checks on an ML model to identify one or more biases, if any, that may be inherent to the ML model is disclosed. Bias assessment results generated by performing the checks are then reported to users, such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes or instructions executable by one or more processors, and the like.

[0005] In one embodiment, the bias assessment functionality is provided as a cloud service by a Bias Assessment as a Service System (BEaaSS). Subscribers to such a cloud service may submit ML models to the service and receive a bias assessment report generated by the service for the submitted ML model. The bias report may include information identifying the bias checks performed by BEaaSS and their results. A recipient of the bias report, such as a data scientist, may use the information in the bias report to reduce any identified biases. For example, a data scientist may modify the training dataset used to train the model to reduce a particular identified bias.

[0006] The bias assessment system embodiments described herein provide an automated solution for detecting and assessing various bias types inherent in a trained ML model. Based on attributes of the ML model and possibly a potentially biased training dataset used to train the ML model, the bias assessment system is programmed to generate one or more synthetic datasets that can be input to the ML model. The trained ML model makes predictions based on the input synthetic dataset, and the predictions can be evaluated for bias. The synthetic dataset includes a set of synthetic data points that, when input to a trained ML model with little or no inherent training bias, predicts no biased predicted data. The same set of synthetic data points, when input to a trained ML model with substantially inherent training bias, results in substantially biased output predicted data. Bias can be assessed by manually inspecting the variability in predictions after introducing artificial bias into the dataset. When the predicted output of the ML model changes, the model may be susceptible to bias. The amount of bias can change based on how much the new output predictions deviate from the expected predictions.

[0007] The predicted data generated by inputting a set of synthetic data points into the trained ML model is processed by an evaluator configured to detect specific bias types in the predicted data. One or more of the evaluations may be performed for one or more specific bias types, the results may be collated in a bias report, and the report may be output to the parties. In some cases, the results and / or reports may be used to perform a number of downstream actions. The results, reports, and actions provide a bias detection and evaluation mechanism through which ML model hosts, users, and other parties may improve model training, testing, and implementation.

[0008] Trained ML models to be evaluated for bias according to embodiments described herein may be received from a number of sources (e.g., user / customer devices, repositories or trained ML models, serverless functions, etc.). In some embodiments, a customer submits the trained ML models to a Bias Evaluation as a Service System (BEaaSS) to determine if the customer models generate prediction data using one or more biases formed. In some embodiments, a model catalog system is configured to receive the trained ML models for storing in the model catalog. The model catalog system may communicate with the BEaaSS to determine if the received models have generated predictions according to an acceptable level of bias prior to including the models in the model catalog.

[0009] In some embodiments, the bias assessment system implements an attribute classifier for determining one or more attributes of the trained ML model and / or the training data used to train the ML model. The attribute classifier is configured to process the ML model data and / or the training data used to train the ML model to determine one or more attributes of the ML model. The bias assessment system may use the one or more determined attributes to generate synthetic data that may be input to the trained ML model to generate predictions.

[0010] The synthetic data generator may generate synthetic data based on one or more determined attributes of the trained ML model. The one or more synthetic datasets may be input to the trained ML model to generate predicted data. The predicted data may be parsed by one or more checking entities in the BEaaSS to determine whether the predicted data corresponds to a type of bias. The one or more synthetic datasets may be generated in some format to attempt to influence the trained ML model to output predicted data with bias. For example, synthetic datasets may be generated with an unbalanced distribution of attribute values ​​at various synthetic data points to attempt to influence the trained ML model to generate and output biased predicted data. In some embodiments, the synthetic datasets may be stored in a synthetic data repository after generation. When a request is received to evaluate a trained ML model for bias, and attributes of the trained ML model share the same attributes as the synthetic datasets stored in the repository, the synthetic dataset may be selected for input to the trained ML model without requiring generation of a new synthetic dataset.

[0011] The bias assessment system may be programmed to perform multiple bias checks in an automated fashion, each of which is a check for bias in the predicted data output by the ML model given one or more synthetic datasets as input. Each bias check corresponds to a different type of bias that may be inherent in the trained ML model. There is no limit to the types of bias that the bias assessment system may assess for a trained ML model, and the assessment may be performed in any combination or fashion useful in assessing ML models.

[0012] The bias assessment system utilizes predicted data generated by a trained ML model using a synthetic dataset input to determine that the trained ML model operates according to one or more biases. In some embodiments, the bias assessment system may supplement the evaluation of the predicted data with an additional evaluation of the training dataset used to train the ML model. For example, a bias assessment entity in the bias assessment system may combine the results of a bias check of the predicted data with the results of a bias check of the training data to generate combined result data.

[0013] The results obtained by the bias assessment system by performing the bias check may then be output along with the predictions made using the ML model. These results provide additional information to consumers of the predictions regarding the probability that the predictions made by the ML model are biased and the corresponding potential impact on the generated predictions. Thus, the bias assessment results generated and output by the bias assessment system may provide a warning system that the model's predictions are biased, that the model itself is improperly trained, or that the training dataset used to train the ML model is flawed.

[0014] Results of the one or more bias assessments for one or more bias types may be compiled and used by the bias assessment system to generate a bias report. The bias report may include report information indicating the tendency of the trained ML model to generate predicted data corresponding to one or more particular biases. The bias report may include report information related to one or more bias checks performed and corresponding bias check results. The bias report may further include an overall bias score indicating the total extent of bias detection in the predicted data and / or one or more downstream actions or recommendations that may be taken as a result of the bias assessment.

[0015] The set of one or more bias check results may be used to determine one or more downstream actions to perform in response to the generation of the one or more bias check results. For example, one downstream action may be to generate and send a message to a data scientist of the enterprise that the trained ML model has a tendency to generate biased predicted data and / or that the trained ML model was trained using a flawed / biased training dataset based on the one or more bias check results. Another downstream action may be to prevent further dissemination of the trained ML model or to prevent inclusion of the trained ML model in a model catalog. Yet another downstream action may be to send the predicted data and a bias report to the client / service that originally supplied the trained ML model.

[0016] The bias assessment system, in some embodiments, may be offered as a cloud service by a cloud service provider, with the service being made available to customers or subscribers who subscribe to this service and other services offered by the cloud service provider.

[0017] In an embodiment, a technique is disclosed in which a bias evaluator as a service system performs processing including: for a trained model to be evaluated, a computing system determines a set of model attributes for the trained model; the computing system generates a first synthetic dataset used for a first bias check performed for the trained model based on the set of model attributes, the first bias check being configured to evaluate the trained model for a first bias type, the first synthetic dataset includes a plurality of data points; using the trained model, generating first prediction data for the first synthetic dataset, the first prediction data includes a first plurality of predicted values ​​generated by the trained model for the plurality of data points in the first synthetic dataset; the computing system generates a first bias result for the first bias type based on the first prediction data; and the computing system generates a bias evaluation report for the trained model, the bias evaluation report including information indicative of the first bias result.

[0018] In an embodiment, the first bias result includes one or more bias values ​​generated based on the first prediction data. In some further embodiments, the bias assessment report includes the first bias result and the one or more bias values, and the method further includes outputting the bias assessment report. In other further embodiments, the method further includes comparing at least the bias value of the one or more bias values ​​to a bias-to-bias threshold, and further includes determining whether to accept or reject the trained model for inclusion in the group of trained models based on the comparison.

[0019] In an embodiment, the process further includes the computing system generating a second synthetic dataset used for a second bias check performed for the trained model based on the set of model attributes, the second bias check being configured to evaluate the trained model for a second bias type, the second synthetic dataset including a plurality of data points, using the trained model to generate a second set of predictions for the second synthetic dataset, and the computing system generating a second bias result for the first bias type based on the first prediction data. In some further embodiments, the method further includes the computing system generating a bias score based on the first bias result and the second bias result, and determining whether to accept or reject the trained model for inclusion in the group of trained models based on the generated bias score.

[0020] In an embodiment, determining the set of model attributes includes processing the trained model to determine at least one model attribute in the set of model attributes. In some embodiments, determining the set of model attributes includes determining at least one model attribute in the set of model attributes based on an analysis of training data used to train and generate the trained model.

[0021] In an embodiment, the process further includes determining training data used to train and generate the trained model, and further includes the computing system generating a second bias result for the first bias type based on the training data, and generating the first bias result is further based on the generated second bias result. In some embodiments, generating the first synthetic dataset includes the computing system generating the first synthetic dataset using a generative neural network machine learning model based on the set of model attributes for the trained model.

[0022] In one embodiment, a system, such as a bias evaluator as a service system, includes a processor and a memory including instructions that, when executed by the processor, cause the device to perform the processes described herein. In another exemplary embodiment, a non-transitory computer-readable medium stores a plurality of instructions executable by one or more processors to cause the one or more processors to perform the processes described herein.

[0023] The above features and aspects, together with other features and aspects, will become more apparent with reference to the following specification, claims and accompanying drawings. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a simplified diagram of a distributed environment incorporating a bias evaluator as a service system, according to various embodiments. [Diagram 2] 4 is a simplified diagram of a bias checker component of a bias evaluator as a service system in accordance with various embodiments. [Diagram 3] FIG. 1 illustrates a simplified diagram of an attribute identifier component of a bias evaluator as a service system, according to various embodiments. [Figure 4] 1 is a simplified flow diagram illustrating an example process for generating a bias assessment for a trained machine learning model using a bias evaluator as a service system, in accordance with various embodiments. [Diagram 5] 1 illustrates an exemplary bias assessment report generated by a bias evaluator as a service system, according to various embodiments. [Figure 6] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 7]FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 8] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 9] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an exemplary computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain aspects. However, it will be apparent that various aspects may be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other aspects or designs.

[0026] The present disclosure relates to assessing bias in trained machine learning (ML) models. A system configured to perform various bias checks on an ML model to identify one or more biases, if any, that may be inherent to the ML model is disclosed. Bias assessment results generated by performing the checks are then reported to users, such as consumers of the ML model, data scientists responsible for modeling and training the ML model, and others. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes or instructions executable by one or more processors, and the like.

[0027] In one embodiment, the bias assessment functionality is provided as a cloud service by a Bias Assessment as a Service System (BEaaSS). Subscribers to such a cloud service may submit ML models to the service and receive a bias assessment report generated by the service for the submitted ML model. The bias report may include information identifying the bias checks performed by BEaaSS and their results. A recipient of the bias report, such as a data scientist, may use the information in the bias report to reduce any identified biases. For example, a data scientist may modify the training dataset used to train the model to reduce a particular identified bias.

[0028] There are various reasons why bias creeps into a trained ML model. Typically, this occurs when the dataset used to train the model contains data points that are heavily represented or weighted towards certain attributes. Such weighted or skewed datasets may result in a trained model that does not adequately represent the real-world environment or use case for which the trained model is used to make predictions. For example, a model trained to select resumes of potential candidates may be biased towards selecting male or female resumes, given that the training dataset used to train the model contains mostly male resumes with very few data points corresponding to female resumes. Trained ML can be biased due to different types of biases in the training dataset, such as sample bias, exclusion bias, measurement bias, label / recall bias, observer bias, association bias, gender bias, religious bias, racial bias, and others. High levels of bias can cause the model to miss relevant relationships between features or attributes of the data and the output predicted by the model.

[0029] If a particular training dataset used to train an ML model contains biased data, the trained ML model is also more likely to make biased predictions. Bias is a common, but non-trivial, problem in ML and is the cause of many erroneous predictions made by ML models. For example, a biased ML model may not only fail to predict correct information, but may also predict incorrect information that the ML model represents as a proper prediction. For example, a trained ML model that generates predictive data in a biased format may be more likely to generate incorrect predictions when certain input data points are provided to the machine learning model. A trained ML model is "biased" when the predictive data generated by the ML model contains undesirable values ​​or outcomes that do not occur in the same proportions as if generated by a correctly trained ML model. Bias occurs when an ML model is trained in a particular format such that the model has a tendency to output biased predictive data. Because training datasets are often not uniform (i.e., the attributes of the data in the datasets used to train the model are often incomplete or biased), trained ML models are often biased due to training on these datasets. For example, a "biased" ML model may tend to predict incorrect values, or values ​​outside the expected range of the base given the input data points. Biased models also vary the output predictions in a manner that is heavily influenced by the input data with high levels of bias in the input dataset. Bias can manifest itself in many types and formats, including, for example, selection bias, stereotyping bias, reporting bias, in-group bias, anecdotal fallacy, etc.

[0030] Biased ML models generate undesirable prediction data more frequently than properly trained ML models. Even worse, the model may be biased towards one prediction for one attribute / type rather than another. Thus, the model may appear to predict data with high accuracy for some input data points, but produce biased prediction data for other, more specific input data points. This creates a problem for services that host trained ML models for customers, and the problem becomes even greater when the service hosts ML models that were not trained under the service provider's supervision. For example, in the case of a service provider that hosts ML models for customers where the service provider is not involved in and has no control over the training of the model, the ML model is like a "black box" to the service provider, who may not have any insight into the potential bias of the ML model. Bias is not a problem that can be solved simply by increasing the size of the training dataset, and a larger biased training dataset may only reinforce undesirable biased behavior in the trained ML model. The "black box" nature of ML models means that bias often cannot be detected by simply investigating the organization of the trained ML model itself.

[0031] The bias assessment system embodiments described herein provide an automated solution for detecting and assessing various bias types that may be inherent in a trained ML model. In an embodiment, the input provided to the bias assessment system for the ML model to be evaluated may be the model to be evaluated and, if available, a training dataset used to train the ML model. The bias assessment system is programmed to determine attributes of the ML model to be evaluated. When the training dataset is provided as an input, the bias assessment system may analyze the training dataset to determine attributes of the input to the model. Based on the identified attributes, the bias assessment system uses synthetic data generation techniques to generate one or more synthetic datasets, each synthetic dataset being directed to checking for a particular bias and including a number of synthetically generated data points to check for that particular bias.

[0032] The bias assessment system then performs various bias checks using the synthetic dataset and generates an assessment result. For a particular bias being assessed, the bias assessment system inputs the synthetic dataset generated for that bias into the ML model being assessed and determines a predicted value generated by the ML for each of the input data points from the synthetic dataset. The predicted values ​​generated by the ML model for the data points in the synthetic dataset for a particular bias check are collectively referred to as predicted data generated by the ML model for the synthetic dataset for the particular bias check. The bias assessment system then analyzes the predicted data to generate a bias assessment result for that particular bias check. In an embodiment, the bias assessment result indicates the degree of bias exhibited by the ML model for the particular bias being assessed. The degree may be indicated by a numerical value, a score, a visualization (e.g., a graph), and the like.

[0033] One example of such an ML model is a trained neural network (NN) consisting of a number of nodes organized into layers, each layer containing one or more nodes. The layers include an input layer to which inputs are provided, and an output layer that outputs predicted values ​​for the inputs provided as inputs to the input layer. Zero or more intermediate layers may be interposed between the input layer and the output layer, with the output of one layer provided as input to the next layer. For such a neural network model, for synthetic data points provided as inputs to the input layer of the neural network, the output layer of the neural network outputs predicted values ​​representing the values ​​predicted by the neural network for the inputs.

[0034] Additionally, as part of generating a predicted value, each of the intermediate layers and nodes in the layer may generate intermediate values ​​that are passed to the next layer until the output layer is reached and the output layer outputs a predicted value. In such an embodiment, the predicted value generated by the model for the synthetic dataset includes the final value predicted by the output layer of the neural network. This predicted data is then analyzed by a bias evaluation system to generate bias results for a particular bias check. As described herein, the final result of the ML model processing the input is referred to as "predicted data."

[0035] For example, once the predicted data is generated, it is processed by an evaluator configured to detect a particular bias type in the predicted data. The evaluation may use the predicted data for the synthetic dataset to evaluate the ML model for a particular bias, and may use other aspects of the bias detection process, such as one or more attributes, input synthetic data points, etc. For example, the bias evaluator may compare the predicted data output by the trained ML model to one or more synthetic input values ​​to determine an expected distribution of predicted values ​​and an actual distribution of predicted values. In some embodiments, the evaluation may be based on statistical decisions and functions applied to the set of predicted data.

[0036] One or more of the evaluations may be performed for one or more specific bias types, and the results may be arranged in a bias report. The report may be output to the parties. For example, for a trained ML model being evaluated, the results of the various bias checks performed for that model may be compiled or aggregated into a comprehensive report reviewable by the submitting entity that submitted the trained ML model for bias evaluation. The report may include information indicating the various bias checks performed by the bias evaluation system and their corresponding evaluation results, an overall bias evaluation result, and a number of suggested downstream actions, such as sending the report to an entity, rejecting the model from the model catalog, retraining the ML model, etc. In some embodiments, a user or entity receiving the bias evaluation report may select one or more downstream actions to perform based on the generated bias report. For example, a service that performs report generation may also include one or more subsystems, whereby a user may indicate that one or more subsystems should be operated. For example, a retraining subsystem for retraining a biased model may be suggested by the bias report, and a user may indicate to the subsystem to retrain a biased model in response to receiving the report, etc.

[0037] In an embodiment, the degree of a particular bias may be represented using a quantification associated with the bias result. A variety of different quantifications may be used, such as scores, graphs, etc., that represent the degree of bias. For example, the results of the various bias checks performed may be compiled in a number of visual graphs and score quantifications that are recognizable and understandable by humans and / or machines.

[0038] In some embodiments, an overall bias score may be displayed in the bias assessment report. The overall bias assessment may be a compiled bias assessment score based on multiple sub-scores corresponding to multiple specific biases assessed. For example, the overall bias assessment score may be an overall score, such as an average score, or multiple sub-scores of the individual bias assessments.

[0039] In some cases, the results and / or reports may be used to perform a number of downstream actions. The downstream actions may include, for example, corrective actions to correct bias, or other responsive actions responsive to the results of the bias evaluation. The corrective actions may be actions to correct biased behavior in the model in response to detection of bias in the results. For example, retraining the ML model with an unbiased training dataset may be a downstream action to correct bias behavior in the ML model. The corrective actions may be initiated by a user in response to receiving the bias report. The responsive actions may be actions to occur based on the state of the model after the evaluation. For example, a decision to reject the model for inclusion in the model catalog and rejection of the model from the model catalog may occur in response to a determination that the ML model operates with an unacceptable level of bias.

[0040] The results, reports, and actions provide a bias detection and evaluation mechanism through which ML model hosts, users, and other parties may improve model training, testing, and implementation. For example, in some embodiments, a model catalog that provides a repository for ML models submitted by users may use the bias evaluation service of BEaaSS. The model catalog may submit a bias evaluation request to BEaaSS, which includes the ML model to be evaluated and may also include training data used to train the model. After performing the bias evaluation, BEaaSS may generate a bias evaluation report and send the report to the model catalog as a response to the request sent by the model catalog. The model catalog may then take one or more actions based on the bias evaluation report. In one implementation, if the report indicates that the ML model failed the bias evaluation, the model catalog may reject the model from being added to the model catalog. The model catalog may allow the ML model to be added to the model catalog only if it receives a passing bias evaluation result from BEaaSS.

[0041] Techniques for facilitating BEaaSS provide numerous technical advantages that improve the function and performance of trained ML models. For example, BEaaSS as described herein provides a novel automated bias assessment system and service that was not previously available. For example, model assessment is primarily performed manually and bias-by-bias. Implementation of BEaaSS as described herein enables automated comprehensive bias assessment for trained ML models. BEaaSS facilitates techniques for automatically ingesting trained ML models, performing comprehensive bias assessment, and outputting results in response to a request. This provides a centralized system for automated model improvement for various bias tests.

[0042] The described BEaaSS also provides technical advantages by removing the action from the client and owner of the trained ML model. Due to the automated nature of BEaaSS, the model owner / model catalog does not need to be actively involved in the model evaluation and bias evaluation. The checks, including the generation of synthetic datasets to perform the checks, are performed automatically by BEaaSS without requiring input from the client beyond the request to evaluate the model. This improves user resource utilization while centralizing the system for evaluating ML models. The user does not need to know what checks are performed or the central mechanism of the checks. Instead, the user only needs to generate a request for model evaluation and wait for the report that is automatically generated in response to the request.

[0043] The described BEaaSS also provides flexible check capabilities that can be updated as needed as a centralized service. For example, as new bias checks are developed, improved, and otherwise modified, the aggregated nature of BEaaSS allows the checks to be updated in one place without the user having to manually download and augment the latest set of bias checks. This makes BEaaSS more easily integrated into users' workflows and also preserves user resources. It also ensures that secure applications such as model catalogs are constantly updated with the most contemporary checks to comprehensively diagnose new problematic biases in near real-time.

[0044] Exemplary Systems and Embodiments FIG. 1 is a simplified diagram of a distributed environment incorporating a bias evaluator as a service system, according to various embodiments. As shown in FIG. 1, the distributed environment includes multiple systems and subsystems. The distributed environment includes a Bias Evaluation as a Service System (BEaaSS) 100. The BEaaSS 100 may be a service implemented in a computing device, such as a server system. The BEaaSS may be a service configured to facilitate the embodiments described herein, i.e., evaluation of a trained ML model for one or more bias types. For example, the BEaaSS may be a service configured to receive a trained ML model from a separate external system, evaluate the trained ML model for bias, and return a bias report and / or perform downstream operations based on the evaluation.

[0045] The controller subsystem 130 is a subsystem of the BEaaSS 100. The controller subsystem 130 may be a subsystem configured to facilitate the operation of the BEaaSS 100 and its communication with external systems. For example, the controller subsystem 130 may be a central subsystem with access to various other subsystems of the BEaaSS 100. The controller subsystem 130 may thus facilitate the transmission of data within the BEaaSS 100 and initiate various other subsystems to perform the embodiments described herein. The controller subsystem 130 may communicate with systems external to the BEaaSS 100 to import and output data as part of the embodiments described herein. For example, the BEaaSS 100 may be implemented as part of a model evaluation service provided by a service provider to customers. The controller subsystem 130 may be configured to communicate with customer / user devices to receive trained ML models for evaluation as part of the service.

[0046] User device 110 may be a computing device or system communicatively coupled to BEaaSS 100, for example, via controller subsystem 130. User device 110 may be a computing device utilized by a customer of a service implementing BEaaSS 100 to send a trained ML model for evaluation. For example, a service provider may offer a service implementing BEaaSS 100 through which a customer / user may submit a trained ML model owned by the customer / user for evaluation. In response to sending the trained ML model to BEaaSS 100, the customer may receive a response including information and / or actions related to the trained ML model.

[0047] The user device 110 may send the model and training data and bias type 111 for evaluation to the controller subsystem 130. The model sent to the controller subsystem 130 is the trained ML model to be evaluated for bias. In some embodiments, the user device may also send training data to the controller subsystem 130. The training data may include at least a training data set used to train the trained ML model, which is also sent to the controller subsystem. In some embodiments, the bias type may also be sent to the controller subsystem 130. The bias type may refer to one or more bias checks that a customer using the user device 110 may have requested as part of the bias evaluation for the trained ML model. In some embodiments where the bias type is not sent, all available bias checks are performed by default on the trained ML model. In some embodiments where the bias type is not sent, a subsystem of the BEaaSS 100, such as an attribute classifier, may process the trained ML model and possibly the training data set to determine one or more bias types that are possible or likely to be specific to the trained ML model.

[0048] Once completed, a bias assessment report 112 may be sent from the controller subsystem 130 to the user device 110 in response to sending the model for evaluation and the training data and bias type 111. More information about bias assessment reports is described below.

[0049] In some embodiments, the BEaaSS 100 is communicatively coupled to a model catalog 120. The model catalog 120 may be a model service that hosts a number of trained ML models that may be utilized by customers of the model service. For example, the model service may allow a source customer / user to upload their trained ML models for inclusion in the model service. Other customers / users may then utilize the source customer / user's trained ML models. As noted above, for this reason, it is important that a model service provider only host trained ML models with little or no inherent bias in order to maintain an optimal model service.

[0050] In some embodiments, the model catalog 120 may also include a domain ontology store for storing a plurality of models, where each model is grouped into a domain. Each domain specifies attributes or industries common to a number of models. A model recommendation engine of the model catalog 120 may compare customer-provided data to a plurality of models in the domain ontology store and identify one or more recommended models for the customer's use. For example, the model recommendation engine may identify a number of similar terms between the parsed data and terms associated with each model. In some examples, the model recommendation engine may generate a confidence metric for each model. The confidence metric may include a value (a percentage from a range of values) indicating an estimated likelihood that the model is relevant to the set of data provided by the customer. The confidence value may be based on the number of common terms between each model and the parsed data from the dataset. The model recommendation engine may identify one or more recommended models specific to the data provided by the customer. For example, a list of recommended models and a description of each recommended model may be provided to a client device for selection of a recommended model. The model catalog 120 may generate a domain ontology store and may continually add new models to the domain ontology store.

[0051] The feedback and learning engine can obtain feedback data from model recommendations and subsequent selection / rejection of models for a particular set of data provided by a customer. As mentioned above, it is important that the models provided to customers generate predictive data with little or no bias. It is a very important goal of the model catalog service provider to maintain the model catalog 120 to only models that contain an acceptable level of bias.

[0052] The model catalog 120 may receive models for submission and training data 121. The models for submission and training data 121 may be received, for example, from a user of the model catalog seeking to submit a model for submission for inclusion in the model catalog. Training data may also be provided along with the model for submission. The model catalog 120 is communicatively coupled to the BEaaSS 100 via the controller subsystem 130. The model catalog 120 may send a request to the BEaaSS 100 to perform an evaluation for the model for submission before accepting the model for submission as part of the model catalog 120.

[0053] The model catalog 120 may send the models 122 for submission and training data and bias types to the controller subsystem 130. The models for submission and training data are the same data received as part of the submission described above. In addition, the model catalog may generate and send to the BEaaSS 100 bias types corresponding to bias checks that the model catalog requires the model for submission to pass before it is accepted into the model catalog. The bias types generated by the model catalog 120 may correspond to some bias checks that are important for a particular model catalog and may exclude others that are not important. For example, a model catalog configured to host models for predicting employee salaries may require that gender, race, and language biases be evaluated to have lower levels than thresholds before the model can be included in them. Other biases, such as foul language, social media presence, etc., may not be important factors for models in the model catalog and therefore are not included in the bias types generated and sent from the model catalog 120.

[0054] Upon completion, a bias assessment report 123 may be transmitted from the controller subsystem 130 to the user device 110. The bias assessment report 123 may be similar to the bias assessment report 112 that may be transmitted to the user device 110 described above. In addition, the bias assessment report 123 may include one or more sets of data corresponding to a recommended decision that the model catalog 120 may take with respect to the submitted model. For example, the bias assessment report may include a recommendation for the model catalog 120 to accept or reject a model for submission for inclusion in the model catalog 120.

[0055] The customer bias evaluation preferences 131 are one or more sets of bias preference configurations stored in the controller subsystem 130. In embodiments where the bias type corresponding to the bias check to be performed is not sent to the BEaaSS 100, the customer bias evaluation preferences 131 may include data regarding one or more preferred bias types for evaluating the trained ML model. For example, a user of the user device 110 may be associated with a customer bias evaluation preference that lists one or more bias types that the user typically uses in trained model bias evaluation. In other examples, the customer bias evaluation preferences 131 stores information regarding an optimal bias type associated with the model catalog 120. As described above, the bias type may determine one or more bias checks to perform for a given trained ML model. Once the trained ML model and bias type are determined / received, the BEaaSS 100 may begin by evaluating the trained ML model for bias.

[0056] The BEaaSS 100 may first determine one or more attributes for generating a synthetic data set that may be input to a trained ML model. The controller subsystem 130 sends the model for evaluation and training data 131 to the attribute identifier 140. The attribute identifier 140 may process the model for evaluation and training data 131 and determine one or more attributes for generating a synthetic data set. The functionality of the attribute identifier 140 is described below with respect to FIG. 3. Once the attributes are determined, the attribute identifier 140 sends the attributes 141 to the controller subsystem 130. In some embodiments, the attribute identifier 140 may send the one or more attributes 141 directly to a synthetic data generator, such as synthetic data generator 150.

[0057] Once one or more attributes for generating synthetic data are determined, BEaaSS 100 generates the synthetic data, which may be input to the trained ML model to generate and output predicted data. The controller subsystem may forward the attributes 141 received from the attribute identifier 140 to the synthetic data generator 150. The synthetic data generator 150 may be a subsystem BEaaSS 100 configured to generate one or more synthetic data sets, which may be input to the trained ML model, thereby causing the trained ML model to generate and output predicted data. The synthetic may be generated such that when the synthetic data sets are input to the one or more trained ML models, the one or more trained ML models are most likely to generate predicted data corresponding to biased predictions. More specifically, the synthetic input data points in the synthetic data sets are generated such that the input data points, when input to the trained ML model, generate predicted data that more clearly demonstrates potential biases inherent in the trained ML model.

[0058] The synthetic data generator 150 may employ one or more ML techniques to generate the synthetic dataset. These techniques may include one or more machine learning (ML) techniques, rule-based techniques, and others. In some implementations, one or more machine learning-based techniques may be used. For example, a generative adversarial network (GAN) may be used to generate synthetic data in the synthetic dataset, where the generated synthetic data closely resembles the original or real data. An example of a GAN architecture is described in "Ian J. Goodfellow et al., Generative Adversarial Nets, NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems, Volume 2, December 2014, pp. 2672-2680". The entire contents of the Goodfellow et al. publication are incorporated herein by reference for all purposes. In some embodiments, generative models such as natural language generation models (NLG) and hidden Markov models (HMMs) and artificial neural networks (ANNs).

[0059] GANs can generate synthetic data based on real data provided as input to the GAN. The synthetic data generated by the GAN resembles real data in terms of required parameters, univariate and multivariate distributions, interrelationships between variables, etc. During training, the GAN learns the true data distribution of the input training dataset with the goal of generating new data points from this distribution with some changes, rather than simply reproducing the old data on which the model was trained. In certain use cases, the synthetic data generated by the GAN can be used to augment the real data and generate a synthetic dataset. Since the GAN may generate data from the distribution of the training data on which the GAN was trained, one or more GANs may be trained on well-known biases, thereby also generating a dataset with these biases. In this way, one or more GANs can be modified to output a biased dataset in a controlled format for use in testing the model for bias. The deep learning model can also modify an existing dataset to generate a modified synthetic dataset that includes the specified bias as part of the bias detection techniques described herein.

[0060] A typical GAN ​​architecture consists of two adversarial models, commonly implemented as neural networks competing with each other. These adversarial models include a generator neural network (generator) and a discriminator neural network (discriminator). The generator is trained to generate new synthetic data based on real data provided as input to the generator. The discriminator is a type of classifier trained to distinguish between real or synthetic data by estimating the probability that the samples generated by the generator are real or generated data. During training of a GAN, the generator and the discriminator play a continuous adversarial game, so that as training progresses, the generator learns to generate more realistic data samples based on the input training data, and the discriminator learns to get better at distinguishing the generated synthetic data from real data. This adversarial cooperation between the two networks is the cause of the success of GANs, as both learn at the expense of each other and reach an equilibrium state over time.

[0061] The trained GAN can then be used to generate synthetic data for data provided as input to the GAN. The GAN may be used to generate various synthetic datasets, for example, as described in this disclosure. There are different GAN architectures for generating different types of synthetic data, including architectures for generating synthetic tabular data. The GAN may be used to generate heterogeneity in the generated synthetic dataset. For example, in an attempt to "trick" a potentially biased ML model, the GAN may generate a synthetic dataset for heterogeneous bias testing (i.e., the synthetic dataset has a varying structure of data attributes, includes missing values, missing data types, etc.). This allows for the introduction of data attributes or data ranges that may not have been present in the training dataset used to train the ML model.

[0062] Other machine learning based techniques besides GANs may be used to generate the synthetic data described in this disclosure. These techniques may include using neural networks (e.g., convolutional neural networks (CNNs)), variational autoencoders (VAEs), decision trees, random forest techniques, linear regression, other deep learning techniques, and others.

[0063] Additionally, non-machine learning based techniques may be used in addition to or instead of machine learning based techniques to generate the synthetic dataset. These include, for example, various sampling and best-fit techniques, Monte Carlo techniques, and others.

[0064] The synthetic data repository 151 may be a repository within the synthetic data generator 150 configured to store the generated synthetic data sets. Once the synthetic data sets are generated by the synthetic data generator 150, the synthetic data sets may be transferred to another subsystem of the BEaaSS 100 and / or stored in the synthetic data repository 151. Storing the generated synthetic data sets in the synthetic data repository 151 may include storing the synthetic data sets according to a particular bias associated with the generated synthetic data set. For example, the synthetic data generator 150 may receive the attributes 141 and a list of bias types including “bias A” to be performed using the generated synthetic data. The synthetic data generator 150 may generate a synthetic data set using one or more attributes to be used in the evaluation of the model for bias A. The synthetic data set for bias A may then be stored in the synthetic data repository 151.

[0065] In some embodiments, the synthetic dataset may be retrieved from the synthetic data repository 151 instead of or in addition to generating new synthetic data. For example, in response to receiving one or more attributes 141 and a list of bias types, the synthetic data generator may search the synthetic data repository 151 for a previously generated synthetic dataset that corresponds to the particular bias type and one or more attributes. If a stored synthetic dataset is found, the stored synthetic dataset may be retrieved and sent to another subsystem of the BEaaSS 100. In some embodiments, techniques such as k-fold validation are used to increase the testing quality of the synthetic dataset generation.

[0066] The synthetic data generator 150 may transmit the synthetic dataset 152 to the controller subsystem 130 and / or the bias checker subsystem 160. In some embodiments, the synthetic data generator 150 transmits the generated synthetic dataset 152 directly to a bias checker subsystem, such as the bias checker subsystem 160. The synthetic dataset 152 may be responsively used by the bias checker subsystem for being an evaluation performance of the trained ML model. In some embodiments, the synthetic data generator 150 transmits the generated synthetic dataset 152 to the controller subsystem 130. The controller subsystem 130 may then responsively group the generated synthetic dataset into a package of data that may be used to perform an evaluation of the trained ML model. For example, the controller subsystem 130 may group the received synthetic dataset 152 with the trained ML model for evaluation, the training data used to train the trained ML model, and bias type data that specifies one or more bias checks to perform. The data may be grouped as a set of check data 132 that is sent to bias checker subsystem 160 to perform one or more bias checks for the trained ML model. Check data 132 may be a comprehensive set of data that includes one or more of synthetic datasets 152, instructions for one or more bias checks to perform, an originating entity of a request to perform a bias assessment, etc.

[0067] Bias checker 160 may be a subsystem of BEaaSS 100 that utilizes one or more bias checkers 161(A)-(N) to perform bias evaluation for various bias types. Bias checker 160 may be configured to receive check data 132 from controller subsystem 130 and / or synthetic data set 152 from synthetic data generator 150 to initiate execution of bias evaluation for a trained ML model. In various embodiments, bias checker subsystem 160 receives a list of bias types and parses the list to determine one or more bias checks to be performed by one or more bias checkers 161. Bias checker subsystem 160 may forward data, such as the trained ML model to be evaluated, the synthetic data, and possibly a training data set, to one or more bias checkers 161 for evaluation of the trained ML model.

[0068] Bias checkers 161(A)-(N) are configured to receive data including a synthetic dataset and a trained ML model for evaluation, and in response generate and output result data 162(A)-(N). Result data 162(A)-(N) corresponds to results of bias evaluation performed on the trained ML model. Further information regarding the functionality of bias checker 161 is described below with reference to FIG. 2.

[0069] The result data 162(A)-(N) generated by the bias checkers 161(A)-(N) is transmitted to one or more subsystems of the BEaaSS 100. For example, the result data 162(A)-(N) is transmitted to the report generator 170. The report generator 170 may be a subsystem of the BEaaSS 100 configured to generate one or more reports including the result data 162(A)-(N) or other information related to the results of the bias assessment. In some embodiments, the report information is compiled into a human and / or computer readable format for processing and determining aspects of the bias assessment. In an embodiment, the overall score generator 171 is a subsystem of the report generator 170 configured to derive a score for the bias assessment process. For example, the overall score generator 171 may utilize the result data 162(A)-(N) and generate an overall score for bias for the trained ML model being evaluated. The report generated by report generator 170, and in some embodiments the overall score generated by overall score generator 171, are sent to a separate entity in report data 172. For example, report data 172 may be returned to user device 110 along with bias assessment report 112 or report data 172 may be returned to model catalog 120 as part of bias assessment report 123. Further details regarding bias assessment reports are described below with reference to FIG.

[0070] The result data 162(A)-(N) may also be sent to the action subsystem 180. The action subsystem 180 may be a subsystem of the BEaaSS 100 configured to take the result data 162(A)-(N) and, in response, cause the execution of one or more actions or generate a recommendation for the execution of one or more actions. The action configuration 181 may be a set of configuration data that specifies how the actions are to be executed based on the result data 162(A)-(N). The action configuration 181 may specify a certain configuration for which action is to be taken if the biased result or biased score meets a numerical threshold for the action. The action subsystem 180 may receive the overall score 173 from the overall score generator 171. The overall score 173 may be parsed and compared to an action threshold to determine whether an action should be taken based on the overall score 173. For example, the action configuration 181 may specify that the action subsystem 180 should send a report to a data scientist if the overall score 173 exceeds a pre-set bias threshold for reporting. In some configurations, the model acceptance rules 182 are a set of rules in the operational subsystem 180. The model acceptance rules 182 specifically describe the criteria for accepting or rejecting a model from the model catalog 120. For example, the model acceptance rules may specify that the operational subsystem 180 sends a recommendation to the model catalog 120 along with the bias assessment report 123 to accept the trained ML model if the bias detected for the model prediction is below an acceptable bias threshold. The operational subsystem 180 may thus perform the operations 183.

[0071] Further details relating to the processing performed by the BEaaSS 100 for generating the bias assessment report provided with the forecast data are described below with reference to FIGS. 2, 3, 4 and 5.

[0072] 2 shows a simplified diagram of the bias checker component of a bias evaluator as a service system, according to various embodiments. As shown in FIG. 2, the illustrated bias checker includes several components through which data is generated. In particular, FIG. 2 shows a bias (A) checker 161(A) for ingesting synthetic data and outputting bias evaluation results.

[0073] As part of the bias checker 160, synthetic data 200 for bias (A) is received by the trained model to be evaluated 210. The model to be evaluated 210 is the trained ML model whose evaluation is sought. In some embodiments, the synthetic data for bias (A) is a set of synthetic data specifically generated to perform bias evaluation for “bias A” in the particular trained model to be evaluated 210, and is generated by the synthetic data generator 150 for that specific purpose. The trained model to be evaluated 210 is configured to receive the synthetic data 200 for bias (A) as input and generate and output predicted data 220 for bias (A). The predicted data 220 for bias (A) may be predicted data including one or more predicted values ​​generated by the trained model to be evaluated 210 using a NN ML model.

[0074] In an exemplary embodiment, bias A is a gender bias and bias test for bias A is a bias test for gender bias in the trained ML model. In the same exemplary embodiment, the trained model 210 being evaluated is a model for predicting an employee's salary given input data points including attribute values ​​such as the employee's name, gender, job, etc. In the same exemplary embodiment, synthetic data 200 for bias (A) is a synthetic data set including several synthetic input data points with attribute values ​​corresponding to a theoretical employee's name, gender, job, etc. Thus, when the synthetic data 200 for bias (A) is input to the trained model being evaluated 210, the trained model outputs predicted data for bias (A) corresponding to several salary predictions for each of the synthetic input data points. In the same exemplary embodiment, the synthetic data 200 for bias (A) may have been generated in a format that indicates bias in the predicted data or bias (A) 220. For example, two different input data points may include similar values ​​across data points with only the exclusion of a value for gender. If dissimilar predicted results are generated for both of the input data points, the model may generate gender biased results.

[0075] The data for bias (A) 220 is sent to a bias (A) evaluator 230. The bias (A) evaluator 230 may be any system, evaluator, score / result generator, or other entity configured to process prediction data and attempt to detect bias for a model. Examples of bias evaluators include Generalized Entropy Index (https: / / doi.org / 10.1145 / 3219819.3220046), Differential Fairness and Bias Amplification (DFBA) (https: / / arxiv.org / pdf / 1807.08362), Wino & gender bias score (http: / / web.cs.ucla.edu / ~kwchang / bibliography / zhao2018gender), LOGAN (http: / / web.cs.ucla.edu / ~kwchang / bibliography / zhao2020logan), and the like.

[0076] The bias (A) evaluator 230 may use any number of techniques to determine the bias outcome for the predicted data. In some embodiments, the bias (A) evaluator 230 compares each predicted outcome value to the corresponding synthetic input data point to determine whether the predicted outcome deviates from the expected predicted outcome. For example, the synthetic data set may be used to determine a statistical range of expected values ​​for the predicted outcome value. The predicted outcome value may then be compared to the statistical range to determine the likelihood that the predicted value was generated according to a certain bias.

[0077] In addition to the prediction data 220 for bias(A), the bias(A) evaluator 220 may be configured to process training data 240 for the trained model. Because the trained model 210 being evaluated was trained using the training data 240 for the trained model, the training data may also include biases inherent to the trained model. For example, the bias(A) evaluator 230 may be configured to extract one or more statistical metrics associated with the training data set to determine whether the training data set corresponds to a bias associated with bias(A).

[0078] The bias(A) evaluator 230 utilizes the bias evaluation performed on the predicted data for bias(A) 220 and the training data for the trained model 240 to generate and output the result for bias(A) 250. In various embodiments, the result for bias(A) 250 is a bias result that may be included in the set of result data 162(A)-(N). The result for bias(A) 250 may correspond to one or more metrics related to the level of bias(A) detected for the trained ML model. For example, the result for bias(A) 250 may include a bias value / score describing the relative level of bias detected for the trained model, evaluated based on the predicted data for bias(A) 220 and, in some embodiments, the training data for the trained model 240.

[0079] 3 illustrates a simplified diagram of an attribute identifier component of a bias evaluator as a service system, according to various embodiments. As illustrated in FIG. 3, the illustrated attribute identifier includes several components through which data is generated. In particular, FIG. 3 illustrates an attribute identifier 140 configured to take in a trained model and a training data set to generate one or more attributes.

[0080] 3, the controller subsystem 130 may send a trained model 300 and / or a training dataset 310 to the attribute classifier 140. The trained model 300 is the trained ML model for which an evaluation for bias is sought, and the training dataset 310 is the dataset on which the trained model was trained.

[0081] The trained model 300 may be received at a trained model scanner 320 of the attribute identifier 140. The trained model scanner 320 may be a subsystem of the attribute identifier 140 configured to parse the trained model 300 to detect one or more attributes of the trained model 300. For example, the trained model scanner 320 may be configured to receive, extract, or otherwise obtain model metadata from the trained model 300 to determine one or more attributes of input data points that may be input to the trained model 300.

[0082] The training data set 310 may be received at a training data scanner 330 of the attribute identifier 140. The training data scanner 330 may be a subsystem of the attribute identifier 140 configured to parse the training data set 310 to detect one or more attributes of the trained model 300. For example, the training data scanner 330 may be configured to parse the training data set 310 to extract one or more column attributes of the training data set 310 that are used to train the trained model 300.

[0083] The results output by the trained model scanner 320 and the training data scanner 330 may be integrated by an attribute aggregator 340 of the attribute identifier 140. The attribute aggregator 340 may be a subsystem of the attribute identifier 140 configured to compile and combine one or more attributes received from the trained model scanner 320 and the training data scanner 330 to form a combined set of one or more attributes. The attribute aggregator 340 may then output the integrated attributes 141 to the synthetic data generator 150.

[0084] With reference to the exemplary embodiment described above, the attribute identifier 140 may receive a trained model 300 that is trained to take in input data points related to employee attributes and output a predicted salary value for the employee. The attribute identifier 140 may send the trained model to the trained model scanner 320, which extracts metadata from the trained model 300 that corresponds to the attributes input to the model. For example, based on the metadata, the trained model scanner may determine attributes such as the employee's name, gender, job function, etc. These attributes are compiled and sent to the synthetic data generator 150 to generate synthetic data having the same attributes.

[0085] FIG. 4 is a simplified flow diagram illustrating an exemplary process for generating bias assessment results for a trained machine learning model using a bias evaluator as a service system, according to various embodiments. The process illustrated in FIG. 4 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The method illustrated in FIG. 4 and described below is intended to be exemplary and non-limiting. Although FIG. 4 illustrates various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the process may be performed in a different order or some steps may be performed in parallel. In an embodiment, such as the embodiment illustrated in FIG. 1, the process illustrated in FIG. 4 may be performed by the BEaaSS 100.

[0086] Process 400 may begin at 402, where BEaaSS 100 receives a request to evaluate an ML model for bias, where the request identifies a trained ML model to be evaluated, where the request optionally includes a training dataset used to train the ML model. The model may be received for any source that may transmit a trained ML model for evaluation, including, for example, user device 110 and / or model catalog 120. The trained model data includes the trained ML model to be evaluated for bias. In some embodiments, the request specifies a location of the trained ML model, such as a URL from which the model may be retrieved. In some embodiments, the training dataset used to train the ML model is received as part of the request, where the training dataset may be further utilized to generate synthetic data for performing one or more bias checks on the trained ML model.

[0087] At 404, a set of model attributes for the trained model is determined. The set of model attributes may be determined by a subsystem of the BEaaSS 100, such as the attribute identifier 140. Determining the set of model attributes may be performed, for example, by extracting metadata from the trained model received at 402. The one or more attributes correspond to one or more categories of inputs accepted by the trained ML model for evaluation. In various embodiments, determining the one or more model attributes for the trained model includes evaluating and / or introspecting the trained ML model. For example, metadata from the trained ML model may be identified and parsed to determine the set of model attributes. In some embodiments where training data is received as part of the request at 402, the training dataset is analyzed to determine the set of model attributes for the trained model.

[0088] Blocks 406-410 of FIG. 4 occur for each bias check performed on the trained model. At 406, a synthetic dataset used for the bias check performed for the trained model is generated based on the set of model attributes determined at 404. The model attributes determined at 404 may be sent to a subsystem of the BEaaSS 100, such as the synthetic data generator 150, to generate the synthetic dataset. The synthetic dataset may be generated according to one or more methods for generating synthetic datasets, such as by using a GAN ML model. In various embodiments, the synthetic dataset is generated based on one or more bias types for which a bias assessment is performed.

[0089] At 408, the synthetic data generated at 406 is input into the trained model received at 402 to generate predicted data for the synthetic dataset. For example, a bias checker, such as the bias checker illustrated in FIG. 2, may be used to input the synthetic data into the trained ML model to generate the predicted data. The set of generated predictions may be output as predicted data corresponding to one or more values ​​generated by inputting one or more synthetic input data points from the synthetic dataset generated at 406 into the trained ML model.

[0090] At 410, a bias result based on the prediction data generated at 408 is generated. The bias result may be evaluated by a bias evaluator, such as the bias evaluator shown in FIG. 2. The bias result may be a value / score for a particular bias processed and output by a particular bias checker or may be a combined set of bias results processed and output by multiple bias checkers. In various embodiments, the bias result is generated by processing the set of predictions generated at 408 and determining one or more statistical distributions of the prediction data based on the synthetic dataset generated at 406. The bias result may thus be based on the relative deviation of the predicted value or set of predictions compared to the statistical distribution generated based on the synthetic dataset to represent a predicted level of bias inherent in the predictions made by the trained ML model.

[0091] In some embodiments, a quantification such as a bias score is calculated based on the bias results generated at 410. For example, the score may be calculated based on the number of input data points in the input synthetic dataset, the number of expected predicted data points in the predicted data, and the actual predicted data points in the predicted data. For example, 1000 synthetic data points may be generated that include a field for gender. The synthetic data may include 500 data points that include a "male" gender and 500 data points that include a "female" gender. The synthetic data is then input into the ML model to generate predicted data. For example, the predicted data may indicate a Boolean value of "true" or "false," and the synthetic data may be generated such that the average of the synthetic dataset includes only differences in the "male" or "female" data fields. The expected of the output predicted dataset may be balanced such that 250 of each of the "male" and "female" data points are expected to correspond to a "true" output and 250 of each of the "male" and "female" data points are expected to correspond to a "false" output. However, the actual output predicted data may indicate a significantly greater proportion of “male” data points to be “true” compared to the “female” data points. This is indicative of gender bias in the ML model. The resulting score may be, for example, the ratio of the number of “female” data points predicted to be “true” to the number of “male” data points predicted to be “true”. For example, if 430 “male” data points are predicted to be “true” and 31 “female” data points are predicted to be “true”, then the bias score may be represented by the formula (1-(FT / MT))·100=92.8. After a bias result is generated for each of the bias checks performed, processing proceeds to 412.

[0092] At 412, an output bias assessment report is generated based on the bias results generated at 410 for each of the bias checks. The bias assessment report may include, for example, the bias results of each particular bias check, a bias score generated based on the bias results, a depiction of the level of bias for the trained ML model, a visual representation of the level of bias detected, etc. The report may be generated, for example, by a report generator 170 configured to generate a bias report for review by an entity. The results, in another example, may be sent to an action subsystem 180 to determine or effect one or more actions that may be taken based on the generated bias results.

[0093] At 414, the bias-assessment report generated at 412 is output. The bias-assessment report may be output to an entity, such as, for example, the entity that issued the request at 402. For example, the report may be generated and output to an entity, such as the user device 110 or the model catalog 120.

[0094] 5 illustrates an exemplary bias assessment report generated by the bias assessment as a service system, according to various embodiments. As shown in FIG 5, the report 500 may be provided to an entity, such as a customer or a data scientist, in a readable format.

[0095] As shown in FIG. 5, the bias assessment report may be provided in a human-readable format. The particular bias assessment report shown in FIG. 5 may correspond to the exemplary embodiment described above relating to a bias detection process for a trained ML model for predicting salary values ​​given input data points related to certain employee characteristics. For example, the report 500 may show the results of at least four different bias checks performed by four different bias checkers 161 for the bias types "gender," "race," "religion," and "language." As shown in FIG. 5, these bias types are presented in the form of score types "WINO," "DFBA," "FRA ...<Custom_Religion> ",<Custom_Language> " is scored by each bias checker. The model being evaluated is shown as "Employee Hiring Model."

[0096] As shown in FIG. 5, the results of the bias check for “gender” bias are displayed with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. Indicators are shown in the visual indicators corresponding to the various levels of bias determined to be present in the trained ML model. As shown in FIG. 5, for Bias Type: Gender, the results of the bias check for gender have detected a bias level that is above “medium” but does not reach “high” for gender bias for the trained ML model being evaluated. This may correspond to a determination that the trained ML model being evaluated tends to predict salary values ​​in a biased manner based on the gender specified by the input data points, but that gender does not affect the predicted salary. As shown in FIG. 5, the bias has a score of “74” out of “100”, which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0097] As used herein, a "score" may be a relative numerical representation of the level of bias detection in a trained ML model as quantified by a bias assessment system. For example, a score of 74 out of 100 may mean that 74 out of 100 results in the predicted data generated by processing the input synthetic dataset contained some determined form of bias. In other embodiments, the score may represent a relative statistical score of the model's bias assessment based on a sampling of similar models. The bar graph shown adjacent to the score field in FIG. 5 may indicate the relative level of bias as measured on a scale of bias detected in the model. For example, a display of the level of gender bias in FIG. 5 above "medium" but not reaching "high" may indicate that the gender bias detected in the model was above the average level of bias in similar models, but was not significantly higher in comparison.

[0098] Also shown in FIG. 5 are the results of the bias check for "race" bias displayed with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. Indicators are present in the visual indicator corresponding to the various levels of bias determined to be present in the trained ML model. As shown in FIG. 5, for bias type: race, the result of the bias check for race, i.e., the bias level, extends beyond "medium" and is slightly short of "high". This may correspond to a determination that the trained ML model being evaluated tends to predict salary values ​​in a biased fashion based on the race specified by the input data points, but that race does not affect the predicted salary. As shown in FIG. 5, bias has a score of "94" out of "100", which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0099] As shown in FIG. 5, the results of the bias check for “religion” bias are displayed with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. Indicators are present in the visual indicator corresponding to various levels of bias determined to be present in the trained ML model. As shown in FIG. 5, for Bias Type: Religion, the results of the bias check for religion indicate that a bias level above “medium” but not reaching “high” has been detected for the religious bias for the trained ML model being evaluated. This may correspond to a determination that the trained model being evaluated tends to predict salary values ​​in a biased manner based on the religion specified by the input data points, but that religion does not affect the predicted salary. As shown in FIG. 5, the bias has a score of “51” out of “100”, which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0100] As shown in FIG. 5, the results of the bias check for "language" bias are displayed with a visual indicator of the determined bias and a bias score on a scale of 1 to 100. Indicators are present in the visual indicator corresponding to the various levels of bias determined to be present in the trained ML model. As shown in FIG. 5, for the bias type: language, the results of the bias check for religion include a relatively low bias level below "medium", but a language bias has been detected for the trained ML model being evaluated. This may correspond to a determination that the trained ML model being evaluated tends to predict salary values ​​in a typically biased manner based on the spoken language specified by the input data points, and that the spoken language does not affect the predicted salary. As shown in FIG. 5, the bias has a score of "13" out of "100", which corresponds to the relative proportion of biased predictions detected during the model prediction operation.

[0101] The report 500 also includes an overall bias assessment visual indicator and an overall combined score. The overall bias assessment score may be an overall score generated by a component such as the overall score generator 171. In some embodiments, the overall score is generated based on the individual scores for the bias types listed above. For example, a combined score of "58" out of "100" may be the overall average of the bias scores for the individual bias result scores. An indicator is present in the visual indicator corresponding to the overall average level of bias detected in the trained ML model. As shown in FIG. 5, the result of the overall bias check for the overall bias is a bias level that exceeds "medium" but does not reach "high". In various embodiments, the overall bias assessment score may be generated as an average score of the other bias scores displayed in the report 500. Any mathematical or statistical combination of the subscores for each bias may be utilized to form an overall score for display in the report. A corresponding bar graph of the overall bias score may indicate the relative level of overall bias for models consisting of each bias type included in the report.

[0102] The report 500 also includes a status indicator of the action to be taken with respect to the trained ML model based on the generated overall score. For example, as shown in FIG. 5, a field indicates that the model status of the trained ML model is “Failed Bias Assessment,” indicating that the evaluated model (“Employee Hiring Model”) contains an unacceptable level of bias. “Failed” may indicate that the model should not be accepted by a catalog entity, such as a model catalog. This may correspond to a determined action to reject the trained ML model from the model catalog 120 due to a determination of an overall bias of at least “medium” for the trained ML model.

[0103] Exemplary Infrastructure-as-a-Service Implementation FIG. 6 illustrates a bias assessment system for determining and reporting trained ML model bias for generating predictions, according to various embodiments. As mentioned above, Infrastructure-as-a-Service (IaaS) is one particular type of cloud computing. IaaS may be configured to provide virtualized computing resources on a public network (e.g., the Internet). In the IaaS model, a cloud computing provider may host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may supply various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) that accompany these infrastructure components. Accordingly, these services may be policy-driven, such that an IaaS user may be able to implement policies to drive load balancing to maintain application availability and performance.

[0104] In some examples, an IaaS customer may access resources and services over a wide area network (WAN), such as the Internet, and can utilize the cloud provider's services to install the remaining elements of the application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can then utilize the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, etc.

[0105] In most cases, the cloud computing model requires the participation of a cloud provider. A cloud provider may be, but need not be, a third-party service that specializes in providing (e.g., offering, renting, selling) IaaS. An entity may choose to deploy a private cloud, in which it becomes its own provider of infrastructure services.

[0106] In some examples, an IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. This may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling the (OS), middleware, and / or application deployment (e.g., self-service virtual machines (that can be spun up on demand)), etc.

[0107] In some instances, IaaS provisioning may mean obtaining computers or virtual hosts for use and installing the necessary libraries or services on them. In most cases, deployment does not include provisioning and may require that provisioning be performed first.

[0108] In some cases, there are two different problems with IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure once everything is provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be solved by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which, how each of them work together) can be described declaratively. In some instances, once the topology is defined, workflows can be generated that generate and / or manage the different components described in the configuration files.

[0109] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potentially on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, there may also be one or more security group rules provisioned to define how the security of the network is set up, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, or the like may also be provisioned. The infrastructure may evolve over time as more infrastructure elements are desired and / or added.

[0110] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques may enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various different geographic locations, sometimes across the globe). However, in some examples, the infrastructure in which the code will be deployed must first be set up. In some examples, provisioning may be done manually, provisioning tools may be utilized to provision resources and / or deployment tools may be utilized to deploy the code once the infrastructure is provisioned.

[0111] 6 is a block diagram 600 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 602 may be communicatively coupled to a secure host tenancy 604, which may include a virtual cloud network (VCN) 606 and a secure host subnet 608. In some examples, the service operator 602 may employ one or more client computing devices, which may be handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and the like, and enabled for Internet, email, short message service (SMS), Blackberry®, or other communications protocols. Alternatively, the client computing devices may be general purpose personal computers including, for example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may be, for example, workstation computers running any of the various commercially available UNIX or UNIX-like operating systems including, without limitation, the various GNU / Linux operating systems such as Google Chrome OS.Alternatively, or in addition, the client computing device may be a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device) and / or any other electronic device, such as a personal messaging device, capable of communicating over a network that can access the VCN 606 and / or the Internet.

[0112] The VCN 606 may include a local peering gateway (LPG) 610, which may be communicatively coupled to a secure shell (SSH) VCN 612 via an LPG 610 included in the SSH VCN 612. The SSH VCN 612 may include an SSH subnet 614, which may be communicatively coupled to a control plane VCN 616 via an LPG 610 included in the control plane VCN 616. The SSH VCN 612 may also be communicatively coupled to a data plane VCN 618 via the LPG 610. The control plane VCN 616 and the data plane VCN 618 may be included in a service tenancy 619, which may be owned and / or operated by the IaaS provider.

[0113] The control plane VCN 616 may include a control plane demilitarized zone (DMZ) tier 620 that functions as a perimeter network (e.g., a portion of an enterprise network between an enterprise intranet and an external network). The DMZ-based servers may have limited liability and may help keep security breaches contained. In addition, the DMZ tier 620 may include one or more load balancer (LB) subnets 622, a control plane app tier 624 that may include an app subnet 626, a control plane data tier 628 that may include a database (DB) subnet 630 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 622 included in the control plane DMZ tier 620 may be communicatively coupled to the app subnet 626 included in the control plane app tier 624 and an Internet gateway 634 that may be included in the control plane VCN 616, and the app subnet 626 may be communicatively coupled to the DB subnet 630, a service gateway 636, and a network address translation (NAT) gateway 638 included in the control plane data tier 628. The control plane VCN 616 may include a service gateway 636 and a NAT gateway 638.

[0114] The control plane VCN 616 can include a data plane mirrored app layer 640 that can include an app subnet 626. The app subnet 626 included in the data plane mirrored app layer 640 can include a virtual network interface controller (VNIC) 642 on which a compute instance 644 can run. The compute instance 644 can communicatively couple the app subnet 626 of the data plane mirrored app layer 640 to the app subnet 626 that can be included in the data plane app layer 646.

[0115] The data plane VCN 618 can include a data plane app layer 646, a data plane DMZ layer 648, and a data plane data layer 650. The data plane DMZ layer 648 can include a LB subnet 622 that can be communicatively coupled to an app subnet 626 of the data plane app layer 646 and an Internet gateway 634 of the data plane VCN 618. The app subnet 626 can be communicatively coupled to a service gateway 636 of the data plane VCN 618 and a NAT gateway 638 of the data plane VCN 618. The data plane data layer 650 can also include a DB subnet 630 that can be communicatively coupled to the app subnet 626 of the data plane app layer 646.

[0116] The Internet gateways 634 of the control plane VCNs 616 and data plane VCNs 618 may be communicatively coupled to a metadata management service 652, which may be communicatively coupled to the public Internet 654. The public Internet 654 may be communicatively coupled to a NAT gateway 638 of the control plane VCNs 616 and data plane VCNs 618. The service gateways 636 of the control plane VCNs 616 and data plane VCNs 618 may be communicatively coupled to cloud services 656.

[0117] In some examples, the service gateways 636 of the control plane VCN 616 and the data plane VCN 618 can make application programming interface (API) calls to the cloud services 656 without traversing the public Internet 654. The API calls from the service gateways 636 to the cloud services 656 can be one-way. The service gateways 636 can make API calls to the cloud services 656, and the cloud services 656 can send the requested data to the service gateways 636. However, the cloud services 656 may not initiate the API calls to the service gateways 636.

[0118] In some examples, secure host tenancy 604 may be directly connected to service tenancy 619, which may otherwise be isolated. Secure host subnet 608 may communicate with SSH subnet 614 through LPG 610, which may allow bidirectional communication on an otherwise isolated system. Connecting secure host subnet 608 to SSH subnet 614 may give secure host subnet 608 access to other entities in service tenancy 619.

[0119] The control plane VCN 616 may enable users of a service tenancy 619 to set up or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 616 may be deployed or otherwise used in the data plane VCN 618. In some examples, the control plane VCN 616 may be isolated from the data plane VCN 618, and a data plane mirror app layer 640 of the control plane VCN 616 can communicate with a data plane app layer 646 of the data plane VCN 618 via a VNIC 642, which may be included in the data plane mirror app layer 640 and the data plane app layer 646.

[0120] In some examples, a user, or customer, of the system may make a request, e.g., create, read, update, or delete (CRUD) operations, over the public Internet 654, which may communicate the request to a metadata management service 652. The metadata management service 652 may communicate the request to the control plane VCN 616 through an Internet gateway 634. The request may be received by a LB subnet 622 included in the control plane DMZ layer 620. The LB subnet 622 may determine that the request is valid, and in response to this determination, the LB subnet 622 may send the request to an app subnet 626 included in the control plane app layer 624. If the request is authenticated and requests a call to the public Internet 654, the call to the public Internet 654 may be sent to a NAT gateway 638 that may make the call to the public Internet 654. Memory that may be desired to be stored with the request may be stored in the DB subnet 630.

[0121] In some examples, the data plane mirror app layer 640 can facilitate direct communication between the control plane VCN 616 and the data plane VCN 618. For example, it may be desired that configuration changes, updates, or otherwise appropriate modifications be applied to resources included in the data plane VCN 618. Through the VNIC 642, the control plane VCN 616 can communicate directly with the resources included in the data plane VCN 618, thereby enabling the control plane VCN 616 to perform configuration changes, updates, or otherwise appropriate modifications to those resources.

[0122] In some embodiments, the control plane VCN 616 and the data plane VCN 618 may be included in the service tenancy 619. In this case, a user or customer of the system may not own or operate either the control plane VCN 616 or the data plane VCN 618. An IaaS provider may own or operate the control plane VCN 616 and the data plane VCN 618, both of which may be included in the service tenancy 619. This embodiment may enable network isolation, which may prevent a user or customer from interacting with resources of other users or other customers. This embodiment may also enable a user or customer of the system to store databases privately, without having to rely on the public Internet 654 for storage, which may not have the desired level of security.

[0123] In other embodiments, the LB subnet 622 included in the control plane VCN 616 may be configured to receive signals from the service gateway 636. In this embodiment, the control plane VCN 616 and the data plane VCN 618 may be configured to be called by the IaaS provider's customers without calling the public Internet 654. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in a service tenancy 619, which may be controlled by the IaaS provider and may be isolated from the public Internet 654.

[0124] 7 is a block diagram 700 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 702 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 704 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network 706 (e.g., VCN 606 of FIG. 6) and a secure host subnet 708 (e.g., secure host subnet 608 of FIG. 6). The VCN 706 may include a local peering gateway (LPG) 710 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to a secure shell (SSH) VCN 712 (e.g., SSH VCN 612 of FIG. 6) via an LPG 610 included in the SSH VCN 712. The SSH VCN 712 can include an SSH subnet 714 (e.g., SSH subnet 614 in FIG. 6), which can be communicatively coupled to a control plane VCN 716 (e.g., control plane VCN 616 in FIG. 6) via an LPG 710 included in the control plane VCN 716. The control plane VCN 716 can be included in a service tenancy 719 (e.g., service tenancy 619 in FIG. 6), and the data plane VCN 718 (e.g., data plane VCN 618 in FIG. 6) can be included in a customer tenancy 721, which can be owned or operated by a user or customer of the system.

[0125] The control plane VCN 716 may include a control plane DMZ tier 720 (e.g., control plane DMZ tier 620 of FIG. 6 ) that may include a LB subnet 722 (e.g., LB subnet 622 of FIG. 6 ), a control plane app tier 724 (e.g., control plane app tier 624 of FIG. 6 ) that may include an app subnet 726 (e.g., app subnet 626 of FIG. 6 ), and a control plane data tier 728 (e.g., control plane data tier 628 of FIG. 6 ) that may include a database (DB) subnet 730 (e.g., similar to DB subnet 630 of FIG. 6 ). The LB subnet 722 included in the control plane DMZ layer 720 may be communicatively coupled to an app subnet 726 included in the control plane app layer 724 and an Internet gateway 734 (e.g., Internet gateway 634 of FIG. 6) that may be included in the control plane VCN 716, and the app subnet 726 may be communicatively coupled to a DB subnet 730 included in the control plane data layer 728, a service gateway 736 (e.g., service gateway of FIG. 6), and a network address translation (NAT) gateway 738 (e.g., NAT gateway 638 of FIG. 6). The control plane VCN 716 may include the service gateway 736 and the NAT gateway 738.

[0126] The control plane VCN 716 may include a data plane mirror app layer 740 (e.g., data plane mirror app layer 640 of FIG. 6 ), which may include an app subnet 726. The app subnet 726 included in the data plane mirror app layer 740 may include a virtual network interface controller (VNIC) 742 (e.g., VNIC 642) on which a compute instance 744 (e.g., similar to compute instance 644 of FIG. 6 ) can run. The compute instance 744 can facilitate communication between the app subnet 726 of the data plane mirror app layer 740 and the app subnet 726 that may be included in the data plane app layer 746 (e.g., data plane app layer 646 of FIG. 6 ) via the VNIC 742 included in the data plane mirror app layer 740 and the VNIC 742 included in the data plane app layer 746.

[0127] An Internet gateway 734 included in the control plane VCN 716 may be communicatively coupled to a metadata management service 752 (e.g., metadata management service 652 of FIG. 6), which may be communicatively coupled to a public Internet 754 (e.g., public Internet 654 of FIG. 6). The public Internet 754 may be communicatively coupled to a NAT gateway 738 included in the control plane VCN 716. A service gateway 736 included in the control plane VCN 716 may be communicatively coupled to cloud services 756 (e.g., cloud services 656 of FIG. 6).

[0128] In some examples, the data plane VCN 718 may be included in the customer tenancy 721. In this case, the IaaS provider may provide a control plane VCN 716 for each customer, and the IaaS provider may set up a unique compute instance 744 included in the service tenancy 719 for each customer. Each compute instance 744 may enable communication between the control plane VCN 716 included in the service tenancy 719 and the data plane VCN 718 included in the customer tenancy 721. The compute instance 744 may enable resources provisioned in the control plane VCN 716 included in the service tenancy 719 to be deployed or otherwise used in the data plane VCN 718 included in the customer tenancy 721.

[0129] In other examples, an IaaS provider customer may have a database that resides in customer tenancy 721. In this example, control plane VCN 716 may include a data plane mirror app layer 740 that may include app subnet 726. The data plane mirror app layer 740 may be in data plane VCN 718, but the data plane mirror app layer 740 may not reside in data plane VCN 718. That is, the data plane mirror app layer 740 may have access to customer tenancy 721, but the data plane mirror app layer 740 may not reside in data plane VCN 718 or be owned or operated by the IaaS provider customer. The data plane mirror app layer 740 may be configured to make calls to the data plane VCN 718, but may not be configured to make calls to any entities included in the control plane VCN 716. A customer may desire to deploy or otherwise use resources in the data plane VCN 718 provisioned in the control plane VCN 716, and the data plane mirror app layer 740 can facilitate the customer's desired deployment or otherwise use of the resources.

[0130] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 718. In this embodiment, the customer can determine what the data plane VCN 718 may have access to, and the customer may limit access from the data plane VCN 718 to the public Internet 754. The IaaS provider may not be able to apply filters or otherwise control the data plane VCN 718's access to any external networks or databases. The application of filters and controls by the customer to the data plane VCN 718 contained in the customer tenancy 721 can help to isolate the data plane VCN 718 from other customers and the public Internet 754.

[0131] In some embodiments, cloud services 756 may be called by service gateway 736 to access services that may not reside on the public internet 754, control plane VCN 716, or data plane VCN 718. The connection between cloud services 756 and control plane VCN 716 or data plane VCN 718 may not be live or continuous. Cloud services 756 may reside in different networks owned or operated by an IaaS provider. Cloud services 756 may be configured to receive calls from service gateway 736 and may be configured not to receive calls from the public internet 754. Some cloud services 756 may be isolated from other cloud services 756, and control plane VCN 716 may be isolated from cloud services 756 that may not be in the same region as control plane VCN 716. For example, control plane VCN 716 may be located in “Region 1” and cloud service “Deployment 8” may be located in Region 1 and “Region 2.” When a call to deployment 8 is made by a service gateway 736 included in a control plane VCN 716 located in region 1, the call may be sent to the deployment 8 in region 1. In this example, the control plane VCN 716 or deployment 8 in region 1 may not be communicatively coupled to or otherwise in communication with the deployment 8 in region 2.

[0132] 8 is a block diagram 800 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 802 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 804 (e.g., secure host tenancy 604 of FIG. 6), which may include a virtual cloud network (VCN) 806 (e.g., VCN 606 of FIG. 6) and a secure host subnet 808 (e.g., secure host subnet 608 of FIG. 6). VCN 806 may include an LPG 810 (e.g., LPG 610 of FIG. 6), which may be communicatively coupled to an SSH VCN 812 (e.g., SSH VCN 612 of FIG. 6) via an LPG 810 included in SSH VCN 812. SSH VCN 812 can include an SSH subnet 814 (e.g., SSH subnet 614 in FIG. 6), which can be communicatively coupled to a control plane VCN 816 (e.g., control plane VCN 616 in FIG. 6) via an LPG 810 included in the control plane VCN 816, and to a data plane VCN 818 (e.g., data plane 618 in FIG. 6) via an LPG 810 included in the data plane VCN 818. The control plane VCN 816 and the data plane VCN 818 can be included in a service tenancy 819 (e.g., service tenancy 619 in FIG. 6).

[0133] The control plane VCN 816 may include a control plane DMZ tier 820 (e.g., control plane DMZ tier 620 of FIG. 6 ) that may include a load balancer (LB) subnet 822 (e.g., LB subnet 622 of FIG. 6 ), a control plane app tier 824 (e.g., control plane app tier 624 of FIG. 6 ) that may include an app subnet 826 (similar to app subnet 626 of FIG. 6 ), and a control plane data tier 828 (control plane data tier 628 of FIG. 6 ) that may include a DB subnet 830. The LB subnet 822 included in the control plane DMZ layer 820 may be communicatively coupled to an app subnet 826 included in the control plane app layer 824 and to an Internet gateway 834 (e.g., Internet gateway 634 in FIG. 6 ) that may be included in the control plane VCN 816, and the app subnet 826 may be communicatively coupled to a DB subnet 830 included in the control plane data layer 828, to a service gateway 836 (e.g., service gateway in FIG. 6 ), and to a network address translation (NAT) gateway 838 (e.g., NAT gateway 638 in FIG. 6 ). The control plane VCN 816 may include the service gateway 836 and the NAT gateway 838.

[0134] The data plane VCN 818 may include a data plane app layer 846 (e.g., data plane app layer 646 of FIG. 6), a data plane DMZ layer 848 (data plane DMZ layer 648 of FIG. 6), and a data plane data layer 850 (e.g., data plane data layer 650 of FIG. 6). The data plane DMZ layer 848 may include a LB subnet 822 that may be communicatively coupled to a trusted app subnet 860 and an untrusted app subnet 862 of the data plane app layer 846 and to an Internet gateway 834 included in the data plane VCN 818. The trusted app subnet 860 may be communicatively coupled to a service gateway 836 included in the data plane VCN 818, a NAT gateway 838 included in the data plane VCN 818, and a DB subnet 830 included in the data plane data layer 850. The untrusted app subnet 862 may be communicatively coupled to a service gateway 836 included in the data plane VCN 818 and a DB subnet 830 included in the data plane data layer 850. The data plane data layer 850 may include a DB subnet 830 that may be communicatively coupled to a service gateway 836 included in the data plane VCN 818.

[0135] The untrusted app subnet 862 may include one or more primary VNICs that may be communicatively coupled to tenant virtual machines (VMs) 866(1)-(N). Each tenant VM 866(1)-(N) may be communicatively coupled to a respective app subnet 867(1)-(N) that may be included in a respective container egress VCN 868(1)-(N) that may be included in a respective customer tenancy 870(1)-(N). Each secondary VNIC 872(1)-(N) may facilitate communication between the untrusted app subnet 862 included in the data plane VCN 818 and the app subnet included in the container egress VCN 868(1)-(N). Each container egress VCN 868(1)-(N) may include a NAT gateway 838 that may be communicatively coupled to the public Internet 854 (e.g., public Internet 654 of FIG. 6).

[0136] An Internet gateway 834 included in the control plane VCN 816 and the data plane VCN 818 may be communicatively coupled to a metadata management service 852 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public Internet 854. The public Internet 854 may be communicatively coupled to a NAT gateway 838 included in the control plane VCN 816 and the data plane VCN 818. A service gateway 836 included in the control plane VCN 816 and the data plane VCN 818 may be communicatively coupled to cloud services 856.

[0137] In some embodiments, data plane VCN 818 may be integrated with customer tenancies 870. This integration may be useful or desirable for an IaaS provider's customer in some cases, such as when they may want support when executing code. A customer may provide code for execution that may be disruptive, communicate with other customer resources, or otherwise produce undesirable effects. In response, the IaaS provider may determine whether to execute the code provided to the IaaS provider by the customer.

[0138] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider and may request a function to be attached to data plane layer app 846. Code to execute the function may be executed in VM 866(1)-(N), and the code may not be configured to execute anywhere else in data plane VCN 818. Each VM 866(1)-(N) may be connected to one customer tenancy 870. Each container 871(1)-(N) included in VM 866(1)-(N) may be configured to execute code. In this case, dual isolation can exist (e.g., containers 871(1)-(N) execute code and containers 871(1)-(N) may be contained in at least VMs 866(1)-(N) contained in untrusted app subnet 862), which can help prevent erroneous or otherwise unwanted code from damaging the IaaS provider's network or damaging a different customer's network. Containers 871(1)-(N) may be communicatively coupled to customer tenancy 870 and configured to send and receive data to and from customer tenancy 870. Containers 871(1)-(N) may not be configured to send and receive data to and from any other entities in data plane VCN 818. Once code execution is complete, the IaaS provider may kill or otherwise discard containers 871(1)-(N).

[0139] In some embodiments, trusted app subnet 860 may execute code that may be owned or operated by the IaaS provider. Trusted app subnet 860 may be communicatively coupled to DB subnet 830 and may be configured to perform CRUD operations on DB subnet 830. Untrusted app subnet 862 may be communicatively coupled to DB subnet 830, but in this embodiment, the untrusted app subnet may be configured to perform read operations on DB subnet 830. Containers 871(1)-(N), which may be included in each customer's VMs 866(1)-(N) and may execute code from the customer, may not be communicatively coupled to DB subnet 830.

[0140] In other embodiments, the control plane VCN 816 and the data plane VCN 818 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 816 and the data plane VCN 818. However, communication may occur indirectly through at least one method. An LPG 810 may be installed by an IaaS provider that can facilitate communication between the control plane VCN 816 and the data plane VCN 818. In another example, the control plane VCN 816 or the data plane VCN 818 may make a call to a cloud service 856 through a service gateway 836. For example, a call from the control plane VCN 816 to the cloud service 856 may include a request for a service that can communicate with the data plane VCN 818.

[0141] 9 is a block diagram 900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 902 (e.g., service operator 602 of FIG. 6) may be communicatively coupled to a secure host tenancy 904 (secure host tenancy 604 of FIG. 6) that may include a virtual cloud network (VCN) 906 (e.g., VCN 606 of FIG. 6) and a secure host subnet 908 (e.g., secure host subnet 608 of FIG. 6). The VCN 906 may include an LPG 910 (e.g., LPG 610 of FIG. 6) that may be communicatively coupled to an SSH VCN 912 (e.g., SSH VCN 612 of FIG. 6) via an LPG 910 included in the SSH VCN 912. The SSH VCN 912 can include an SSH subnet 914 (e.g., SSH subnet 614 in FIG. 6), which can be communicatively coupled to a control plane VCN 916 (e.g., control plane VCN 616 in FIG. 6) via an LPG 910 included in the control plane VCN 916, and to a data plane VCN 918 (e.g., data plane 618 in FIG. 6) via an LPG 910 included in the data plane VCN 918. The control plane VCN 916 and the data plane VCN 918 can be included in a service tenancy 919 (e.g., service tenancy 619 in FIG. 6).

[0142] The control plane VCN 916 may include a control plane DMZ layer 920 (e.g., control plane DMZ layer 620 of FIG. 6) that may include a LB subnet 922 (e.g., LB subnet 622 of FIG. 6), a control plane app layer 924 (e.g., control plane app layer 624 of FIG. 6) that may include an app subnet 926 (e.g., app subnet 626 of FIG. 6), and a control plane data layer 928 (e.g., control plane data layer 628 of FIG. 6) that may include a DB subnet 930 (e.g., DB subnet 830 of FIG. 8). The LB subnet 922 included in the control plane DMZ tier 920 may be communicatively coupled to an app subnet 926 included in the control plane app tier 924 and an Internet gateway 934 (e.g., Internet gateway 634 of FIG. 6) that may be included in the control plane VCN 916, and the app subnet 926 may be communicatively coupled to a DB subnet 930 included in the control plane data tier 928, a service gateway 936 (e.g., service gateway of FIG. 6), and a network address translation (NAT) gateway 938 (e.g., NAT gateway 638 of FIG. 6). The control plane VCN 916 may include the service gateway 936 and the NAT gateway 938.

[0143] The data plane VCN 918 may include a data plane app layer 946 (e.g., data plane app layer 646 of FIG. 6), a data plane DMZ layer 948 (e.g., data plane DMZ layer 648 of FIG. 6), and a data plane data layer 950 (e.g., data plane data layer 650 of FIG. 6). The data plane DMZ layer 948 may include a LB subnet 922 that may be communicatively coupled to a trusted app subnet 960 (e.g., trusted app subnet 860 of FIG. 8) and an untrusted app subnet 962 (e.g., untrusted app subnet 862 of FIG. 8) of the data plane app layer 946 and an Internet gateway 934 included in the data plane VCN 918. The trusted app subnet 960 may be communicatively coupled to a service gateway 936 included in the data plane VCN 918, a NAT gateway 938 included in the data plane VCN 918, and a DB subnet 930 included in the data plane data layer 950. The untrusted app subnet 962 may be communicatively coupled to a service gateway 936 included in the data plane VCN 918 and a DB subnet 930 included in the data plane data layer 950. The data plane data layer 950 may include the DB subnet 930, which may be communicatively coupled to a service gateway 936 included in the data plane VCN 918.

[0144] The untrusted app subnet 962 may include primary VNICs 964(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 966(1)-(N) that reside within the untrusted app subnet 962. Each tenant VM 966(1)-(N) may execute code in a respective container 967(1)-(N) and may be communicatively coupled to an app subnet 926 that may be included in a data plane app layer 946 that may be included in a container egress VCN 968. Each secondary VNIC 972(1)-(N) may facilitate communication between the untrusted app subnet 962 included in the data plane VCN 918 and the app subnet included in the container egress VCN 968. The container egress VCN may include a NAT gateway 938 that may be communicatively coupled to the public Internet 954 (e.g., public Internet 654 of FIG. 6).

[0145] An internet gateway 934 included in the control plane VCN 916 and the data plane VCN 918 may be communicatively coupled to a metadata management service 952 (e.g., metadata management system 652 of FIG. 6 ), which may be communicatively coupled to the public internet 954. The public internet 954 may be communicatively coupled to a NAT gateway 938 included in the control plane VCN 916 and the data plane VCN 918. A service gateway 936 included in the control plane VCN 916 and the data plane VCN 918 may be communicatively coupled to cloud services 956.

[0146] In some examples, the pattern illustrated by the architecture of block diagram 900 of FIG. 9 may be considered an exception to the pattern illustrated by the architecture of block diagram 800 of FIG. 8 and may be desirable for a customer of an IaaS provider when the IaaS provider cannot directly communicate with the customer (e.g., in a disconnected region). Each container 967(1)-(N) included in VM 966(1)-(N) for each customer may be accessed in real time by the customer. The containers 967(1)-(N) may be configured to make calls to each secondary VNIC 972(1)-(N) included in app subnet 926 of data plane app tier 946, which may be included in container egress VCN 968. The secondary VNIC 972(1)-(N) may send the call to NAT gateway 938, which may send the call to the public Internet 954. In this example, containers 967(1)-(N) that may be accessed in real time by a customer may be isolated from control plane VCN 916 and may be isolated from other entities included in data plane VCN 918. Containers 967(1)-(N) may be isolated from resources from other customers.

[0147] In another example, a customer may use containers 967(1)-(N) to call cloud service 956. In this example, the customer may execute code in containers 967(1)-(N) that requests a service from cloud service 956. Containers 967(1)-(N) may send the request to secondary VNICs 972(1)-(N), which may send the request to a NAT gateway, which may send the request to public internet 954. Public internet 954 may send the request to LB subnet 922 included in control plane VCN 916 via internet gateway 934. In response to determining that the request is valid, the LB subnet may send the request to app subnet 926, which may send the request to cloud service 956 via service gateway 936.

[0148] It should be appreciated that the IaaS architectures 600, 700, 800, 900 depicted in the drawings may have other components than those depicted. Additionally, the embodiments depicted in the drawings are only some examples of cloud infrastructure systems that may incorporate certain embodiments. In some other embodiments, the IaaS systems may have more or fewer components than depicted in the drawings, may combine two or more components, or may have a different configuration or arrangement of components.

[0149] In one embodiment, the IaaS system described herein may include a suite of application, middleware and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available and secure manner. One example of such an IaaS system is Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0150] 10 illustrates an exemplary computer system 1000 that may be used to implement various embodiments. System 1000 may be used to implement any of the computer systems described above. As shown in the drawing, computer system 1000 includes a processing unit 1004 that communicates with a number of peripheral subsystems via a bus subsystem 1002. These peripheral subsystems may include a processing acceleration unit 1006, an I / O subsystem 1008, a storage subsystem 1018, and a communication subsystem 1024. The storage subsystem 1018 includes a tangible computer readable storage medium 1022 and a system memory 1010.

[0151] Bus subsystem 1002 provides a mechanism for allowing the various components and subsystems of computer system 1000 to communicate with each other as intended. Although bus subsystem 1002 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1002 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0152] The processing unit 1004, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1000. One or more processors may be included in the processing unit 1004. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1004 may be implemented as one or more independent processing units 1032 and / or 1034, with each processing unit including a single or multi-core processor. In other embodiments, the processing unit 1004 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0153] In various embodiments, the processing unit 1004 may execute various programs in response to program code and may maintain multiple simultaneously executing programs or processes. At any time, some or all of the program code being executed may reside in the processor 1004 and / or in the storage subsystem 1018. With appropriate programming, the processor 1004 may provide various functions as described above. The computer system 1000 may additionally include a processing acceleration unit 1006, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0154] The I / O subsystem 1008 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, a motion sensing and / or gesture recognition device such as a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and voice instructions. User interface input devices may also include an eye gesture recognition device such as a Google Glass® blink detector that detects eye movements from a user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0155] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, eye tracking devices. In addition, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, medical ultrasound devices, etc. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0156] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, and the like. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1000 to a user or to another computer. For example, user interface output devices may include a variety of display devices that visually convey text, graphics, and audio / video information, such as, but not limited to, monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0157] Computer system 1000 may include a storage subsystem 1018 that includes software elements currently shown as located in system memory 1010. System memory 1010 may store program instructions that are loadable and executable on processing unit 1004, as well as data generated during the execution of these programs.

[0158] Depending on the configuration and type of computer system 1000, the system memory 1010 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 1004. In some implementations, the system memory 1010 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 1000, such as during start-up, may typically be stored in ROM. For example, and without limitation, the system memory 1010 also illustrates application programs 1012, program data 1014, and an operating system 1016, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like. For example, operating system 1016 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, and the like), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 6 OS, and Palm® OS operating systems.

[0159] The storage subsystem 1018 may also provide a tangible computer-readable storage medium for storing the basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by the processor, provide the functionality described above may be stored in the storage subsystem 1018. These software modules or instructions may be executed by the processing unit 1004. The storage subsystem 1018 may also provide a repository for storing data used in accordance with the present disclosure.

[0160] The storage subsystem 1018 may also include a computer readable storage medium reader 1020 that may be further connected to a computer readable storage medium 1022. Together, and optionally in combination with the system memory 1010, the computer readable storage medium 1022 may comprehensively represent remote, local, fixed and / or removable storage devices and media for containing, storing, transmitting and retrieving computer readable information on a temporary and / or more permanent basis.

[0161] The computer readable storage medium 1022 containing the code or portions of code may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, Electronically Erasable Programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or other tangible computer readable medium. This may include non-tangible computer readable media, such as data signals, data transmission, or any other medium that can be used to transmit the desired information and that can be accessed by the computing system 1000.

[0162] For example, the computer readable storage medium 1022 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, magnetic disk drives that read from or write to removable non-volatile magnetic disks, and optical disk drives that read from or write to removable non-volatile optical disks, such as CD ROMs, Blu-Ray® disks, or other optical media. The computer readable storage medium 1022 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD disks, digital video tapes, and the like. The computer readable storage medium 1022 may also include solid state drives (SSDs) based on non-volatile memory, such as flash memory-based SSDs, enterprise flash drives, solid state ROMs, and the like, SSDs based on volatile memory, such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer system 1000.

[0163] The communications subsystem 1024 provides an interface to other computer systems and networks. The communications subsystem 1024 serves as an interface for receiving data from other systems and transmitting data from the computer system 1000 to other systems. For example, the communications subsystem 1024 may enable the computer system 1000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1024 may include modern data network technologies (e.g., using cellular technology, 3G, 4G, or EDGE (Enhanced data rates for global evolution), WiFi (IEEE 802.11 family of standards), or other mobile communications technologies, or any combination thereof), radio frequency (RF) transceiver components for accessing wireless voice and / or data networks, global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communications subsystem 1024 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0164] In some embodiments, the communications subsystem 1024 may receive incoming communications in the form of structured and / or unstructured data feeds 1026, event streams 1028, event updates, and the like for one or more users who may be using the computer system 1000.

[0165] For example, the communications subsystem 1024 may be configured to receive real-time data feeds 1026 from users of social networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0166] Additionally, the communications subsystem 1024 may be configured to receive data in the form of continuous data streams, which may include event streams 1028 of real-time events and / or event updates 1030, which may be continuous or infinite in nature with no clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0167] The communications subsystem 1024 may be configured to output structured and / or unstructured data feeds 1026, event streams 1028, event updates 1030, and the like to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 1000.

[0168] The computer system 1000 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system.

[0169] Due to the ever-changing nature of computers and networks, the description of the computer system 1000 shown in the drawings is intended as a specific example only. Many other configurations are possible having more or fewer components than the system shown in the drawings. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other means and / or methods for implementing the various embodiments.

[0170] Although specific embodiments have been described, various modifications, variations, alternative structures and equivalents are within the scope of the disclosure. The embodiments are not limited to operation in one particular data processing environment, but are free to operate in multiple data processing environments. In addition, while the embodiments are described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that the claims are not limited to the described sequence of transactions and steps. Various features and aspects of the above-described embodiments may be used individually or jointly.

[0171] Furthermore, while the embodiments are described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the disclosed embodiments. The embodiments may be implemented in hardware only, in software only, or using a combination thereof. The various processes described herein may be implemented in the same processor or different processors in any combination. Thus, where a component or module is described as being configured to perform an operation, such configuration may be realized, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0172] Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. It will be apparent, however, that additions, subtractions, deletions and other modifications and changes may be made thereto without departing from the broader spirit and scope as set forth in the appended claims. Thus, while specific embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the embodiments described in the appended claims.

[0173] Use of the terms "a," "an," and "the" and similar referents in the context of describing the disclosed embodiments (in the context of the claims below) should be construed to include both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., including, but not limited to), unless otherwise stated. The term "connected" should be construed as partially or wholly contained, attached, or joined, even if there is something intervening. The recitation of ranges of values ​​herein is only intended to serve as a shorthand method of referring individually to each separate value contained in the range, and each separate value is incorporated into the specification as if it were individually recited herein, unless otherwise indicated herein or otherwise clearly contradicted by context. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. Any and all examples or exemplary language (e.g., "etc.") provided herein are intended only to further clarify certain embodiments and do not limit the scope of the disclosed technology. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the claimed embodiment.

[0174] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as being generally used to indicate that an item, term, etc. may be X, Y, or Z, or any combination thereof (e.g., X, Y and / or Z), unless specifically stated otherwise. Thus, disjunctive language is not generally intended, and should not imply, that an embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0175] Preferred embodiments, including the best modes known for carrying out various embodiments, are described herein. Variations of these preferred embodiments may become apparent to those skilled in the art upon reading the above description. Those skilled in the art should be able to adopt such variations as appropriate, and the described embodiments may be implemented in other ways than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by this disclosure unless otherwise indicated herein.

[0176] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0177] Although novel aspects are described in this specification with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described embodiments may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method executed by a computer, comprising: for a trained model to be evaluated, a computing system determines a set of model attributes for the trained model; the method further comprises: the computing system generates a first synthetic dataset for use in a first bias check performed for the trained model based on the set of model attributes, the first bias check being configured to evaluate the trained model with respect to a first bias type, the first synthetic dataset including a plurality of data points; the method further comprises: generating first prediction data for the first synthetic dataset using the trained model, the first prediction data including a first plurality of predicted values generated by the trained model for the plurality of data points in the first synthetic dataset; the method further comprises: the computing system generates a first bias result for the first bias type based on the first prediction data; the method further comprises: the computing system generates a bias evaluation report for the trained model, the bias evaluation report including information indicating the first bias result, a method executed by a computer.

2. The method executed by a computer according to claim 1, wherein the first bias result includes one or more bias values generated based on the first prediction data.

3. The method executed by a computer according to claim 2, wherein the bias evaluation report includes the first bias result and the one or more bias values, and the method further comprises outputting the bias evaluation report.

4. The method further comprises: comparing at least one of the one or more bias values with a bias-to-bias threshold; determining whether to accept or reject including the trained model in a group of trained models based on the comparison, the method executed by a computer according to claim 2.

5. The method further comprises: The computing system further includes generating a second synthetic dataset for use in a second bias check to be performed for the trained model based on the set of model attributes, the second bias check being configured to evaluate the trained model with respect to a second bias type, the second synthetic dataset including a plurality of data points, The method includes, using the trained model to generate a second set of predictions for the second synthetic dataset, The method according to claim 1, further comprising the computing system generating a second bias result for the first bias type based on the first prediction data.

6. The method includes, the computing system generating a bias score based on the first bias result and the second bias result, The method according to claim 5, further comprising determining whether to accept or reject including the trained model in a group of the trained models based on the generated bias score.

7. Determining the set of model attributes includes processing the trained model to determine at least one model attribute in the set of model attributes, the method according to claim 1, executed by a computer.

8. Determining the set of model attributes includes determining at least one model attribute in the set of model attributes based on an analysis of training data used to train and generate the trained model, the method according to claim 1, executed by a computer.

9. The method includes, determining training data used to train and generate the trained model, The method according to claim 1, further comprising the computing system generating a second bias result for the first bias type based on the training data, generating the first bias result including generating further based on the generated second bias result.

10. Generating the first synthetic data set includes the computing system using a generative neural network machine learning model to generate the first synthetic data set based on the set of model attributes for the trained model, the method executed by a computer according to claim 1. **Claim 11** The trained model is a neural network, The method according to claim 1, wherein the prediction data further includes at least one value generated by an output layer of the neural network. **Claim 12** One or more processors, A system comprising: a memory including instructions that, when executed by the one or more processors, cause the one or more processors to execute the method according to any one of claims 1 to 11. **Claim 13** A program for causing one or more processors to execute the method according to any one of claims 1 to 11.

Images