Determining bias in trained neural networks using generative neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining a measure of bias for a trained neural network One of the methods includes obtaining a plurality of initial network inputs, wherein each of the initial network inputs has been classified as belonging to a respective ground truth class, and wherein each of the initial network inputs is associated with a corresponding feature value of a particular feature; processing each of the plurality of initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input; determining, for each class of the respective ground truth classes for the initial network inputs, a respective effect size; and determining a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth classes.

Description

[0001]DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application DETERMINING BIAS IN TRAINED NEURAL NETWORKS USING GENERATIVE NEURAL NETWORKS CROSS-REFERENCE TO RELATED APPLICATION This application claims the benefit of the filing date of U.S. Application No. 63 / 586,412, filed on September 28, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND This specification relates to performing a machine learning task on a network input using neural networks. Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters. SUMMARY This specification describes a system implemented as computer programs on one or more computers in one or more locations that determines bias for a trained neural network. “Bias” as used in this specification can indicate a spurious association between the trained neural network’s predictions for given inputs and a feature associated with the given inputs. The system can determine a measure of skewness for the trained neural network. The measure of skewness, also referred to as the measure of bias, can be used to determine whether to pause training of the trained neural network, or further train the trained neural network, or whether to deploy the trained neural network for use. In one aspect there is described a computer-implemented method obtains a plurality of initial network inputs, each of which has been classified as belonging to a respective ground truth class. Each of the initial network inputs is associated with a corresponding feature value of a particular feature. In general a feature can be any characteristic of the neural network inputs, such as a descriptive label of an aspect of one or more of the initial neural network inputs, as an example the color of an object in an DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application image. The corresponding feature value is one of a plurality of feature values for the particular feature. That is the feature value of an initial neural network input can be a value of the feature for that particular initial neural network input. The method involves processing each of the plurality of initial network inputs using a trained neural network, referred to as a target neural network, to generate a respective predicted network output for each initial network input, the respective predicted network output defining a respective predicted class for the initial network input, e.g. a category of a set of multiple categories that the initial network input belongs to. The method determines a respective effect size that represents an association between each class and the particular feature. The effect size can be determined for each class of the respective ground truth classes for the initial network inputs. Alternatively the effect size can be determined for each of the predicted classes for the initial network inputs. For example the method can involve, for each particular class of the respective ground truth (or predicted) classes, identifying initial network inputs for which the respective ground truth class is the particular class, and identifying the predicted network outputs, e.g. the predicted classes, for the identified initial network inputs (finding the initial network inputs for which the ground truth class is the predicted class). The method determines a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth (or predicted) classes for the initial network inputs. In some implementations the initial network inputs can be obtained by generating input representations that describe a ground truth class and a corresponding feature value and processing each of the input representations using a generative neural network to generate one or more initial network inputs conditioned on the input representation. According to one aspect there is provided a method performed by one or more computers, the method comprising: obtaining a plurality of initial network inputs, wherein each of the initial network inputs has been classified as belonging to a respective ground truth class, and wherein each of the initial network inputs is associated with a corresponding feature value of a particular feature, wherein the corresponding feature value is one of a plurality of feature values for the particular feature; processing each of the plurality of initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input, the respective predicted network output defining a respective predicted class for the initial network DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application input; determining, for each class of the respective ground truth classes for the initial network inputs, a respective effect size that represents an association between the class and the particular feature; and determining a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth classes for the initial network inputs. In some implementations, the trained target neural network processes network inputs that represent measurements captured by one or more sensors. In some implementations, the measurements comprise images captured by one or more camera sensors. In some implementations, the measurements comprise audio captured by one or more audio sensors. In some implementations, the measurements comprise observations captured by one or more sensors that sense an environment being interacted with by an agent. In some implementations, the trained target neural network has been trained on a first subset of training data, and obtaining a plurality of initial network inputs that have been classified as belonging to a respective ground truth class comprises: obtaining the plurality of initial network inputs from a second subset of the training data. In some implementations, obtaining a plurality of initial network inputs, wherein each of the initial network inputs has been classified as belonging to a respective ground truth class comprises: generating one or more input representations that describe a ground truth class of the respective ground truth classes and a corresponding feature value of the plurality of feature values; and for each of the one or more input representations, processing the input representation using a first generative neural network to generate one or more initial network inputs that are conditioned on the input representation. In some implementations, each input representation comprises a respective text prompt that describes the ground truth class and the corresponding feature value and wherein the first generative neural network has been trained to generate network outputs that are described by input text prompts. In some implementations, each input representation is generated using a language model that is configured to generate a text sequence that comprises a text prompt that describes a ground truth class and a corresponding feature value, a query about the ground truth class, and an answer that corresponds to the ground truth class, and wherein the input representation includes the text prompt. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some implementations, each initial network input comprises the network output generated using the first generative neural network, and the query of the text sequence generated using the language model. In some implementations, the first generative neural network has been trained to generate network outputs that comprise images that are described by input text prompts. In some implementations, each initial network input comprises an image, and the query for each initial network input comprises a text question about the image for each initial network input. In some implementations, the initial network input comprises an image, and the trained target neural network has been trained to generate a predicted network output that comprises a predicted class for the image. In some implementations, the trained target neural network has been trained to generate a predicted network output that is a text description of a predicted class for the initial network input. In some implementations, processing each of the plurality of initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input comprises: determining that the text description in the network output semantically matches a description of one of the respective ground truth classes for the initial network inputs; and setting the predicted class for the initial network input to be the ground truth class. In some implementations, the method further comprises: determining that the measure of bias is over a threshold measure of bias; and in response to determining that the measure of bias meets a threshold measure of bias, further training the trained target neural network. In some implementations, the method further comprises: determining that the measure of bias is over a threshold measure of bias during training of the trained target neural network. In some implementations, the method further comprises: in response to determining that the measure of bias meets a threshold measure of bias during training of the trained target neural network, halting training of the trained target neural network. In some implementations, the method further comprises: determining to deploy a previous checkpoint of the trained target neural network, wherein the previous checkpoint of the trained target neural has a measure of bias that does not meet the threshold measure of bias. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some implementations, the method further comprises determining that the measure of bias for the trained target neural network represents a higher amount of bias than one or more respective measures of bias for one or more earlier checkpoints of the trained target neural network. In some implementations, the method further comprises: in response to determining that the measure of bias for the trained target neural network represents a higher amount of bias than one or more of the respective measures of bias for one or more earlier checkpoints of the trained target neural network, deploying a checkpoint of the one or more earlier checkpoints of the trained target neural network. In some implementations, further training the trained target neural network comprises: obtaining training data that the trained target neural network was trained on; modifying the training data with respect to the particular feature; and further training the trained target neural network on the modified training data. In some implementations, modifying the training data with respect to the particular feature comprises obtaining additional training data. In some implementations, modifying the training data with respect to the particular feature comprises generating training data using one or more neural networks. In some implementations, generating training data using one or more neural networks comprises de-biasing prompts to the one or more neural networks. In some implementations, further training the trained target neural network comprises using guidance to train the trained target neural network. In some implementations, further training the trained target neural network comprises using an adversarial loss to train the trained target neural network. In some implementations, further training the trained target neural network comprises fine-tuning the trained target neural network. In some implementations, further training the trained target neural network comprises unlearning a subset of training data that the trained target neural network was trained on. In some implementations, the method further comprises: determining whether to deploy the trained target neural network for use in processing network inputs received from users based on the measure of bias. In some implementations, the method further comprises: in response to determining to deploy the trained target neural network, deploying the trained neural network for use in processing network inputs received from users. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some implementations, the method further comprises: receiving a request to deploy the trained target neural network from a requester; and in response to determining not to deploy the trained target neural network, automatically notifying one or more monitors of the request of the determination not to deploy the trained target neural network. In some implementations, automatically notifying the one or more monitors of the request of the determination not to deploy the trained target neural network comprises sending an email to the one or more monitors. In some implementations, automatically notifying the one or more monitors of the request of the determination not to deploy the trained target neural network comprises sending a push notification to the one or more monitors. In some implementations, the method further comprises: receiving, from a user, a user input that represents the particular feature. In some implementations, the method further comprises: providing the measure of bias to a user. In some implementations, the trained target neural network is one of a plurality of trained target neural networks, and wherein the method further comprises: determining a respective measure of bias for each of the trained target neural networks; and identifying a trained target neural network that has a respective measure of bias that represents a lowest amount of bias among the respective measures of bias. In some implementations, the method further comprises deploying the identified trained target neural network. In some implementations, determining, for each class of the respective ground truth classes for the initial network inputs, a respective effect size comprises: for each particular class of the respective ground truth classes: identifying initial network inputs for which the respective ground truth class is the particular class; identifying the predicted network outputs for the identified initial network inputs; for each class of the respective ground truth classes, determining, for each feature value, a respective count of identified initial network inputs that are associated with the feature value and for which the predicted network output defines the class; and determining the effect size for the particular class based on the respective counts of identified initial network inputs for the feature values and the respective predicted classes. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some implementations, determining the effect size for the particular class comprises computing a square root of a chi-squared statistic for the respective counts for the feature values divided by a count of identified predicted network outputs multiplied by a number of degrees of freedom, wherein the number of degrees of freedom is based on a number of respective predicted classes or a number of feature values. In some implementations, the chi-squared statistic is derived from a sum over each respective count for the feature values and the respective predicted classes of a squared difference between the respective count and a corresponding expected count divided by the corresponding expected count. In some implementations, the corresponding expected count is obtained from the plurality of initial network inputs. In some implementations, determining a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth classes for the initial network inputs comprises deriving the measure of bias from each of the respective effect sizes for each of the respective ground truth classes and a mean of the respective effect sizes for each of the respective ground truth classes. In some implementations, deriving the measure of bias from each of the respective effect sizes for each of the respective ground truth classes and a mean of the respective effect sizes for each of the respective ground truth classes comprises computing a sum over each respective ground truth class of a cubed difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes, divided by a sum over each respective ground truth class of a squared difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes raised to a power of 3 / 2. According to another aspect there are provided one or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the methods described herein. According to another aspect there is provided a system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the methods described herein. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Machine learning models such as neural networks can learn unintended biases from correlations in training data. These correlations may be spurious, undesirable, and / or vary across environments. Existing techniques to evaluate unintended biases in model predictions quantify numbers of correct and incorrect predictions for inputs. However, these existing metrics techniques treat as equivalent all incorrect predictions and thus do not provide granularity for investigating how the model distributes incorrect predictions over different feature values of the inputs. For example, a model may consistently misclassify an input blue square as a circle, and an input red square as a triangle. Existing metrics would aggregate the misclassifications across the predicted classes, i.e., shapes, and would not consider how different the distribution of predicted classes are for different feature values, i.e., colors. The system described in this specification can determine a measure of bias for a trained neural network with respect to a particular feature, e.g., color. For example, to determine the measure of bias for the trained neural network, the system can process network inputs using the trained neural network to generate a prediction of a class for each network input. The system can determine an effect size corresponding to the relationship between each ground-truth class and the particular feature. The system can aggregate the effect sizes for each ground-truth class to determine a measure of bias for the particular feature. Furthermore, by determining an effect size for each class, the system can allow for pinpointing classes for which the model has potentially learned an unintended association with a particular feature. For example, although a model may predict a particular class with a high accuracy, the system can use the per-class effect size to determine if the model has potentially learned an unintended association between the particular class and a particular feature. A high accuracy indicates that the model has learned predictive features for the particular class. A high effect size for the particular class indicates that the model makes incorrect predictions differently across different feature values for the particular feature. In cases where a model predicts a particular class with a high accuracy and the effect size for the class is high, this can indicate that when predictions are incorrect, the model relies on unintended associations between the particular class and the particular feature such that incorrect predictions are different for each feature value. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application By using the techniques described in this specification, biases in a trained neural network, e.g., a large-scale vision model or other model that processes sensor data captured by one or more networks, can be automatically discovered under real-world settings without existing datasets for evaluating biases. The techniques described in this specification can discover biases in models for which datasets do not include the ground- truth class and the feature value for each data point. In particular, the described techniques can use off-the-shelf, large-scale, generative models, e.g., image-to-text and text-to-image models, to automatically find biases. For example, the described techniques can use existing models to generate network inputs to provide to the model to be tested for bias. As a particular example, the system can generate one or more input representations that describe a ground truth class and a feature value. The system can process each representation using a generative neural network to generate one or more initial network inputs that are conditioned on the input representation. Thus, the described techniques can effectively discover biases without requiring any additional model training. The techniques described in this specification can also save computing time and resources. For example, the techniques do not require collecting datasets that have known ground truth classes and feature values to be used for testing the model for bias. In addition, the techniques can also save computing time and resources devoted to training. For example, the techniques can determine a measure of bias for a neural network during the training process. If the measure of bias meets a threshold measure of bias, the techniques can stop or halt the training process. The techniques can also use the measure of bias to determine to further train or fine-tune the model in order to address the bias. For example, upon determining that the measure of bias meets a threshold measure of bias, the techniques include determining to deploy a previous checkpoint of the model that does not meet the threshold measure of bias, or training the neural network on modified, e.g., more diverse, training data. The techniques can also provide for the prevention of deploying models for use that may be biased. For example, the system can determine whether to deploy a model for use based on the measure of bias. For example, if the measure of bias meets a threshold measure of bias, the techniques can determine not to deploy the model. The techniques can also provide for the comparison of different trained neural networks or different checkpoints for a trained neural network for selecting a trained neural network or checkpoint with the lowest amount of bias. For example, the system DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application can determine a respective measure of bias for each of multiple trained target neural networks. The system can identify and deploy the trained target neural network that has a respective measure of bias that represents the lowest amount of bias among the respective measures of bias. As another example, the system can determine a measure of bias at each of multiple checkpoints during training of the trained target neural network. In response to determining that the measure of bias represents a higher amount of bias than respective measures of bias for previous checkpoints, the system can determine to deploy a previous checkpoint of the trained target neural network that has a respective measure of bias that represents a lower amount of bias. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 is a block diagram of an example system for determining bias for a trained target neural network. FIG. 2 is a block diagram of another example system for determining bias for a trained target neural network. FIG.3 is a flow diagram of an example process for determining bias for a trained target neural network. FIG.4 is a flow diagram of an example process for determining an effect size for a class. FIG. 5 shows the performance of an example system for determining bias for a trained target neural network. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG. 1 is a block diagram of an example system 100 for determining bias for a trained target neural network 110. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The system 100 is a system that determines a measure of bias 150 for the trained target neural network 110. The trained neural network 110, also referred to as the trained target neural network 110, can be configured to generate a predicted network output given an initial network input, and the network output can define a predicted class for the initial network input. For example, the trained neural network 110 can be a neural network that has been trained to perform a classification task. As used in this specification, a classification task is any task that requires the trained target neural network 110 to generate a network output that defines a predicted class for an initial network input. The class for an initial network input is a category of a set of multiple categories that the initial network input belongs to. Example categories are described below. In some examples, the network output can define a predicted class using a natural language text label for the predicted class of the initial network input. For example, the trained neural network 110 can be a neural network that has been trained to generate a text description of a predicted class for the initial network input. For example, the initial network input can include a data item such as text, an image, a video, or an audio signal. As a particular example, the trained neural network 110 can be configured to provide an answer to a text question about a given image. In some examples, the trained target neural network 110 can determine a respective score for each of a set of multiple classes. The trained target neural network 110 selects one or more of the classes as a “classification” for the network input using the respective scores and generates a network output that includes an identifier for the selected class such as a natural language text label for the selected class. One example of a classification task is image classification, where the initial network input to the trained target neural network 110 is an image, i.e., the intensity values of the pixels of the image, the classes are object categories, and the task is to classify the image as depicting an object from one or more of the object categories. That is, the classification for a given input image is a prediction of one or more object categories that are depicted in the input image. For example, the categories can represent different natural objects that can be depicted in the image. As another example, the images can be medical images and the categories can represent different types of tissue that can be depicted in the image or can represent different states of one or more medical conditions. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application Another example of a classification task is text classification, where the initial network input to the trained target neural network 110 is text and the task is to classify the text as belonging to one of multiple categories. One example of such a task is sentiment analysis task, where the categories each correspond to different possible sentiments of the task. Another example of such a task is a reading comprehension task, where the input text includes a context passage and a question and the categories each correspond to different segments from the context passage that might be an answer to the question. Other examples of text processing tasks that can be framed as classification tasks include an entailment task, a paraphrase task, a textual similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on. Other examples of classification tasks include audio processing tasks where the initial network input to the trained target neural network 110 is audio data. Some examples of audio processing tasks are speech processing tasks, where the input to the neural network is audio data representing speech. Examples of speech processing tasks include language identification (where the categories are different possible languages for the speech), hotword identification (where the categories indicate whether one or more specific “hotwords” are spoken in the audio data), and so on. Other examples of audio processing tasks are tasks that require processing audio data other than speech, e.g., audio event classification (where the categories are different types of audio events that can be audible in the audio data), environmental sound classification (where the categories represent different animals that can make the sound audible in the audio data, or, more generally, different objects that can generate the sound audible in the audio data). As another example, the task can be a health prediction task, where the initial network input is a sequence derived from electronic health record (EHR) data for a patient and the categories are respective predictions that are relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient. As another example, the task can be an agent control task, where the initial network input is an observation captured by one or more sensors that sense an environment being interacted with by an agent and the output is a respective score for each action in a set of actions that can be performed by the agent. The system then selects an action to be performed by the agent by selecting one of the actions using the respective scores. Examples of environments and observations are described below. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The trained target neural network 110 can have any of a variety of architectures suitable for performing a classification task. For example, the trained target neural network 110 can include any appropriate types of neural network layers (e.g., self- attention layers, fully connected layers, convolutional layer, and so forth) in any appropriate numbers (e.g., 5 layers, 10 layers, or 50 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers). In some examples, the trained target neural network 110 can include a state space model (SSM). As a particular example, the trained target neural network 110 can include a recurrent neural network (RNN) or a convolutional neural network (CNN). In some examples, the trained target neural network 110 can include one or more linear recurrent units. As another example, the trained target neural network 110 can have any of a variety of Transformer-based neural network architectures, e.g., encoder-only Transformer architectures, encoder-decoder Transformer architectures, decoder-only Transformer architectures, other attention-based architectures, and so on. In general a Transformer-based architecture can be one which is characterized by having a succession of self-attention neural network layers. A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over attention layer inputs to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used. As a particular example, the trained target neural network 110 can be a language model neural network or visual language model neural network. Some example target neural networks with which the techniques described herein may be used include: BLIP (Li et al. arXiv:2201:12086); Flamingo (Alayrac et al. arXiv:2204.14198); PaLI (Chen et al. arXiv:2209.06794); and PaLI-X (Chen et al. arXiv:2305.18565). The trained neural network 110 can have been trained on a set of training data using an objective function that is appropriate for a classification task, e.g., a cross- entropy loss or other classification loss. That is, the described techniques are generally applicable to determining bias for any conventionally trained neural network. The system 100 can determine a measure of bias 150 of the trained target neural network 110. In particular, the system 100 determines the measure of bias 150 with respect to a particular feature. A feature has multiple possible feature values. A data item such as an image, video, text, or audio can be associated with a feature value if the data DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application item depicts or includes the feature value. For example, the particular feature can be the color of an object depicted in an image, and the feature values can each be a color such as “red,” “blue,” or “green.” An image that depicts a green square can be associated with the feature value of “green.” As another example, the particular feature can be the background depicted in an image, and the feature values can include types of backgrounds such as “rocks,” “sky,” or “road.” More specifically, to determine a measure of bias 150 with respect to the particular feature, the system 100 obtains multiple initial network inputs 102a-n for the trained target neural network 110. Each initial network input can include any of a variety of data items to be classified, such as text, image data, video data, or audio data. In some examples, the initial network inputs can represent measurements captured by one or more sensors. For example, the measurements can include images captured by one or more camera sensors, audio captured by one or more audio sensors, or observations captured by one or more sensors that sense an environment being interacted with by an agent. Each of the initial network inputs 102a-n is associated with a corresponding feature value 106a-n of a particular feature, also referred to as a corresponding attribute. For example, the initial network input 102a is associated with corresponding feature value 106a. In the example of FIG.1, the initial network input 102a includes an image that depicts a sock on a road. The corresponding feature value 106a is “road.” Each of the initial network inputs 102a-n has been classified as belonging to a respective ground truth class 104a-n. As an example, the class can be the type of object depicted in the image. For example, the initial network input 102a has been classified as belonging to ground truth class 104a. In the example of FIG.1, the ground truth class 104a can include “sock.” In some examples, the system 100 obtains the initial network inputs 102a-n from a set of training data that includes the initial network inputs 102a-n, the corresponding feature values 106a-n, and the ground truth classes 104a-n. In these examples, the system 100 determines the measure of bias 150 with respect to the particular feature of the corresponding feature values 106a-n. In some examples, the system 100 generates the initial network inputs 102a-n as described in further detail below with reference to FIG.2. In these examples, the system can receive an input from a user specifying the particular feature. The system 100 can maintain the corresponding feature value and ground truth class for each of the generated DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application initial network inputs 102a-n. Thus the system 100 determines the measure of bias 150 with respect to the particular feature specified by the user. More specifically, to determine a measure of bias 150 with respect to a particular feature, the system processes each of the initial network inputs 102a-n using the trained target neural network 110 to generate respective predicted network outputs 112a-n. As described above, each predicted network output defines a predicted class for the initial network input. For example, the predicted network output 112a defines the predicted class for the initial network input 102a. For example, if an initial network input includes an image, the class can include the shape of an object depicted in the image, a type of object depicted in the image, or an occupation of a subject depicted in the image. The trained target neural network 110 can be configured to generate a predicted network output that includes a predicted class for the data item of the initial network input. In some examples, the trained target neural network 110 can be configured to answer a query about a class of a data item. For example, each initial network input 102a-n can include a query about the class. For example, if the initial network input 102a-n includes an image, the initial network input 102a-n can also include a query about the image, such as a text question about the class of the image. For example, the query can include “What is this object?” In the example of FIG.1, each image can depict an object and a background. The trained target neural network 110 can be configured to classify the type of object depicted in the image. The predicted network output 112a can identify the predicted class “running shoes.” Other predicted classes for the initial network inputs 102a-n can include, for example, “sock” and “stocking.” In some examples, the system 100 can modify the predicted class for one or more of the initial network inputs based on the network outputs. For example, the network output can include a text description of a predicted class for an initial network input. In some examples, the text description can include synonyms of a ground truth class. For example, one of the ground truth classes can include “sock,” and the text description in a network output can include “ankle sock.” As another example, the text description can include answers with different structures. For example, a text description in a network output can include “The object is a sock,” and the text description in another network output can include “A sock.” The system 100 can thus set the predicted class in these DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application examples to be “sock,” so that the system 100 can more accurately determine the effect size for the class. To modify a predicted class, the system 100 can determine that the text description in the network output semantically matches a description of one of the respective ground truth classes for the initial network inputs. For example, the system 100 can use a language model neural network to determine whether the text description semantically matches a description of one of the respective ground truth classes. In response, the system 100 can set the predicted class for the initial network input to be the ground truth class. To determine the measure of bias 150, the system 100 determines, for each class of the ground truth classes, a respective effect size 122. In statistics “effect size” has a particular meaning that generally refers to a value measuring the strength of a relationship between two variables. The respective effect size 122 for each class represents an association between the class and the particular feature, and can comprise a score that represents this association. For example, a higher effect size can indicate that predictions of the class are more affected by changes in the feature value than for a lower effect size. For example, the system 100 can use an effect size engine 120 to determine each effect size 122. An example of determining each effect size 122 is described in further detail below with reference to FIG.4. The system 100 determines the measure of bias 150 of the trained target neural network 110 with respect to the particular feature by aggregating the respective effect sizes 122 over each of the classes. In implementations the measure of bias 150 can be a score, in particular a score that captures how different the distribution of predicted classes are across the feature values for the particular feature. In the example of FIG.1, the system 100 determines the measure of bias 150 with respect to the background feature. For example, the system 100 can use a bias determination engine 130 to determine the measure of bias 150. An example of determining the measure of bias 150 is described in further detail below with reference to FIG.3. In some examples, the system 100 can provide the measure of bias 150 to a user. For example, the system 100 can provide data representing the measure of bias 150 for display on a user device of the user. In some settings the measure of bias can be used to determine a level of trust that can be given to the network output, e.g. predicted class. For example where a medical image or EHR data is processed by the trained (target) DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application neural network a clinician can judge the weight that should be given to the network output in the context of other medical factors. Although this specification describes an example of initial network inputs with images of objects, where the class is the type of object and the feature is the type of background, the initial network inputs can include any of a variety of data items, and the class and feature can be any of a variety of classes and features. As another example, the initial network inputs can include images of objects represented by different shapes, colors, and at different positions, where the class is shape, and the feature is color. FIG.2 is a block diagram of the example system 100 described with reference to FIG.1. In the example of FIG.2, the system 100 also includes a generative neural network 210 and a training engine 240. The system obtains the initial network inputs 102a-n as described above with reference to FIG.1. In some examples, the system obtains the initial network inputs 102a-n from a set of training data. For example, the trained target neural network 110 can have been trained on a first subset of the training data, and the system can obtain the initial network inputs 102a-n from a second subset of the set of training data. As an example, the set of training data can include the initial network inputs 102a- n, the corresponding feature values 106a-n, and the ground truth classes 104a-n. The trained target neural network 110 can have been trained on the initial network inputs and ground truth classes of a first subset of the training data, and the system can obtain the initial network inputs 102a-n, the corresponding feature values 106a-n, and the ground truth classes 104a-n from a second subset of the training data. In some examples, the system generates the initial network inputs 102a-n. For example, the system can use the generative neural network 210, also referred to as the first generative neural network, to generate the initial network inputs 102a-n. In these examples, the system can maintain the corresponding feature value and ground truth class for each of the generated initial network inputs 102a-n. To generate the initial network inputs 102a-n, the system can generate one or more input representations 208 that each include natural language text that describes a ground truth class of the respective ground truth classes and a corresponding feature value of the multiple feature values. In some examples, the system 100 can obtain the corresponding feature value and ground truth class for the initial network inputs 102a-n from the input representations 208. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application For each of the input representations, the system 100 can process the input representation 208 using the generative neural network 210 to generate one or more initial network inputs that are conditioned on the input representation 208. As an example, the system 100 can generate each initial network input for an input representation 208 by sampling from the generative neural network 210 given the input representation 208. In some implementations, the generative neural network 210 can be configured to generate network outputs that are described by input text prompts. The generative neural network 210 can have any appropriate neural network architecture for generating network outputs that are described by input text prompts. For example, the generative neural network 210 can include a language model neural network such as PALM, described in Chowdhery, et al., PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311, PaLM 2, described in Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023), Gemma, described in Gemma Team, et al., Gemma: Open Models Based on Gemini Research and Technology, arXiv preprint arXiv:arXiv:2403.08295 (2024), and Chinchilla, described in Hoffmann et al., Training Compute-Optimal Large Language Models, arXiv:2203.15556 (2022), or a multimodal language model neural network such as Gemini, described in Gemini Team, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). As another example, the generative neural network 210 can include a Transformer-based neural network such as Muse, described in Chang et al., Muse: Text-To-Image Generation via Masked Generative Transformers, arXiv preprint arXiv:2301.00704 (2023). As another example, the generative neural network 210 can include a text-conditioned diffusion model such as Imagen, described in Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, arXiv preprint arXiv:2205.11487 (2022). As an example, each input representation 208 can include a respective text prompt that describes the respective ground truth class and the corresponding feature value. Referring to FIG.1, the system 100 can generate the initial network input 102a by providing an input representation that includes the text “a sock on a road” or “an image of a sock on a road” as input to the generative neural network 210. In some implementations, the system can generate each input representation 208 using a language model neural network. For example, the system can obtain each input representation 208 using a language model neural network that is configured to generate a text sequence that includes a text prompt that describes a ground truth class and a DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application corresponding feature value, a query about the ground truth class, and an answer that corresponds to the ground truth class. In some examples, the text sequence can include a respective text identifier for the text prompt, the query, and the answer. For example, the respective text identifier can include “T2I prompt:”, “Question:”, or “Answer:”. The system can extract the text prompt that describes the ground truth class and the corresponding feature value from the text sequence to generate the input representation 208. For example, the system can use the text identifier for the text prompt to extract the text prompt. In some examples, the system 100 can obtain the ground truth class for the initial network inputs from the text sequence. For example, the system can use the text identifier for the answer to extract the text prompt. In some examples, the language model neural network can be trained to generate the text sequence that includes a text prompt that describes a ground truth class and a corresponding feature value, a query about the ground truth class, and an answer that corresponds to the ground truth class. In some examples, the system 100 can few-shot prompt the language model neural network using a few-shot prompt template to generate the text sequence. The few-shot prompt template can include a text prompt such as “T2I prompt: An image of a shoe on the beach,” a query about the ground truth class such as “Question: What is this object?” and an answer that corresponds to the ground truth class such as “Answer: Shoe.” In some examples, the system can generate the initial network inputs 102a-n based on a list of potential feature values. For example, the system can obtain the list of the feature values from a dataset of feature values. In some examples, the system can generate the initial network inputs 102a-n based on user inputs. For example, the system can receive data representing potential values from a user, e.g., through a user interface of a user device. In some examples, the system can receive the particular feature from a user. In these examples, the system can generate, using a language model neural network, for example, possible feature values for the particular feature to include in different input representations. In some examples, the system can receive data representing one or more classes from a user. For example, classes can include “sock,” “shoe,” or “salamander.” In these examples, the system 100 can include each of the one or more classes in the input representations as ground truth classes. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some examples, the system can receive data representing a type of class from a user. For example, a type of class can include “object” or “shape.” In these examples, the system 100 can generate, using a language model neural network, for example, possible classes to include in the input representations as ground truth classes. The system 100 can thus use the language model neural network to generate a diverse set of text prompts for generating initial network inputs with different ground truth classes and feature values. The system 100 can process the input representations using the generative neural network 210 to generate the initial network inputs 102a-n. For example, the system can include each network output of the generative neural network 210 in an initial network input. In some examples, each initial network input for the trained target neural network 110 can also include a query about the network output of the generative neural network 210. In the example of FIG.2, the generative neural network 210 can be configured to generate network outputs that include images that are described by input text prompts. For example, the generative neural network 210 can be trained to perform an image generation task. Each initial network input for the trained target neural network 110 includes an image that was generated by the generative neural network 210. The system can also include the query of the text sequence generated by the language model neural network described above in the initial network input. The system can process the initial network inputs, such as the initial network input 102a, using the trained target neural network 110, to generate the predicted network outputs 112a-n as described above with reference to FIG.1. The system generates the measure of bias 150 using the effect size engine 120 and the bias determination engine 130 as described above with reference to FIG.1. In some implementations, the system can perform one or more additional actions using the measure of bias 150. For example, the system can determine to further train or to deploy, i.e. use, or make available for use, the trained target neural network 110 based on the measure of bias 150. For example, the system 100 can determine whether the measure of bias 150 meets a threshold measure of bias. In some examples, the threshold measure of bias can be a default value. In some examples, the threshold measure of bias is met if the measure DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application of bias 150 is greater than the threshold measure of bias. In response, the system 100 can further train the trained target neural network 110. For example, the system 100 can further train the trained target neural network 110 using a training engine 240. The training engine 240 is configured to train the trained target neural network 110, e.g., by updating the current parameter values of the neural network at each of multiple training iterations. In some examples where the trained target neural network 110 was trained on a first subset of training data, the training engine 240 can further train the trained target neural network 110 on a second subset of the training data. In some examples, the training engine 240 can further train the trained target neural network 110 on modified training data to mitigate the bias. For example, the training engine 240 can obtain training data that the trained target neural network was trained on. The training engine 240 can modify the training data with respect to the particular feature. As an example, the training engine 240 can obtain additional training data, for example, from a training dataset. As a particular example, the training engine 240 can obtain additional training data that fits certain criteria. For example, during training of a target neural network 110 on the task of image classification, the training engine 240 can identify one or more seed images for which the target neural network 110 performs poorly. The training engine 240 can identify images that are perceptually similar to the seed images from a dataset that includes images and labels for the images. For example, a perceptually similar image has a low distance between the image embedding and the seed image embedding. The training engine 240 can further train the trained target neural network on the identified images and their labels. As another example, the training engine 240 can generate training data to include in the modified training data. For example, the training engine 240 can use one or more neural networks such as the generative neural network 210 to generate the training data. In some examples, the training engine 240 can generate training data by de-biasing prompts to the one or more neural networks. For example, the training engine 240 can include an instruction to be unbiased to the generative neural network 210 when generating its network output. The training engine 240 can further train the trained target neural network on the modified training data. In some examples, the training engine 240 can further train the trained target neural network 110 using training techniques such as guidance, using an adversarial loss, DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application or fine-tuning the trained target neural network. The adversarial loss can be one that maximizes the likelihood of the trained target neural network predicting a correct class while minimizing the ability of another, adversarial network to predict a corresponding feature value from activations in a layer, e.g. an output layer of the trained target neural network. For example, the training engine 240 can train the trained target neural network 110 using guidance or supervised learning. As an example, the training engine 240 can train the target neural network 110 using a supervised learning algorithm that causes the target neural network 110 to learn a feature representation for a given input that is informative for the classification task and uninformative for any features. The training engine 240 can train the target neural network 110 to minimize a joint loss function based on a primary loss function for the classification task, a secondary loss function for the features, and a secondary confusion loss function for the features. The secondary loss function for the features assesses the amount of information related to the features in the feature representation, and the training engine 240 uses the secondary confusion loss function for the features to update the feature representation to remove the information. An example supervised learning algorithm is described in further detail in Alvi et al., Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018. As another example, the training engine 240 can train the trained target neural network 110 using adversarial learning, e.g., using an adversarial network. As a particular example, the adversarial network can be trained to predict a feature value conditioned on the output of a layer of the trained target neural network 110, e.g., the output of the softmax layer. The trained target neural network 110 can be trained to maximize the likelihood of predicting the correct class while minimizing the ability of the adversarial network to predict the feature value. For example, the gradient of the adversarial network can be incorporated into the weight updating rule of the target neural network 110. An example of using an adversarial network to train the trained target neural network 110 is described in Zhang et al., Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI / ACM Conference on AI, Ethics, and Society, pp.335–340, 2018. As another example, the training engine 240 can fine-tune the trained target neural network 110, e.g., on different or modified training data. The training engine 240 can DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application fine-tune some or all of the weights of the trained target neural network 110, e.g., using a different optimizer or on a different objective. As a particular example, the training engine 240 can keep the trained values of some of the parameters fixed while adjusting the trained values of others of the parameters of the trained target neural network 110. In some examples, the training engine 240 can further train the trained target neural network 110 by unlearning a subset of training data that the trained target neural network 110 was trained on, using any suitable machine unlearning technique. In some examples where the trained target neural network 110 includes a language model neural network or visual language model neural network, the training engine 240 can modify the prompts to the trained target neural network 110 to de-bias the text embeddings of the trained target neural network 110. For example, the training engine 240 can use a projection matrix to de-bias the text embedding. The training engine 240 can define a set of biased directions in the text embedding using prompts that describe the biases. The projection matrix is a linear transformation of the text embedding that projects out the biased directions. For example, the projection matrix can be the orthogonal complement of the embeddings of the prompts. In some examples, the training engine 240 can calibrate the projection matrix using a calibration loss that minimizes the discrepancy of a pair of prompt embeddings that have the same semantic meaning after projection. Examples of de-biasing text embeddings are described in further detail in Chuang et al., Debiasing Vision-Language Models via Biased Prompts, arXiv preprint arXiv:2302.00070, 2023. In some implementations, the system 100 can determine whether the measure of bias 150 meets the threshold measure of bias during training of the target neural network 110. For example, the system 100 can determine the measure of bias 150 for the target neural network 110 at each of multiple training iterations during the training of the target neural network 110. As an example, the system 100 can determine the measure of bias 150 at each of multiple training checkpoints. As a particular example, each training checkpoint can be a predetermined number of training iterations after the preceding training checkpoint (or, for the first training checkpoint. In some examples, the system 100 can determine that the measure of bias 150 for the current target neural network 110, i.e., for the current parameter values of the target neural network 110, is over the threshold measure of bias. In response, the system 100 can halt training of the trained target neural network 110. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In some of these examples, the system 100 can determine to deploy a previous checkpoint that has a measure of bias that does not meet the threshold measure of bias. For example, the system 100 can maintain data representing the measure of bias and the network parameter values for previous checkpoints. In response to determining that the measure of bias 150 for the target neural network 110 meets the threshold measure of bias, the system 100 can identify a previous checkpoint that has a measure of bias that does not meet the threshold measure of bias. The system can thus deploy the previous checkpoint of the target neural network 110. In some of these examples, the system 100 can determine to deploy a previous checkpoint that has a measure of bias that represents a lower amount of bias than the current trained target neural network 110. For example, the system can determine that the measure of bias for the current target neural network 110 represents a higher amount of bias than one or more respective measures of bias for one or more earlier checkpoints of the trained target neural network 110. For example, the system can determine that the measure of bias for the current target neural network 110 is greater than a measure of bias for an earlier checkpoint. In response to determining that the measure of bias for the current trained target neural network represents a higher amount of bias than one or more of the respective measures of bias for one or more earlier checkpoints of the trained target neural network, the system 100 can identify a checkpoint of the one or more earlier checkpoints of the trained target neural network that has a measure of bias that represents a lower amount of bias. The system 100 can deploy the identified checkpoint. In some of these examples, the system 100 can determine to restart training of the trained target neural network from a previous checkpoint that has a measure of bias that represents a lower amount of bias than the current trained target neural network 110. For example, the system 100 can identify an earlier checkpoint of the trained target neural network that has a measure of bias that represents a lower amount of bias as described above. The system 100 can further train the trained target neural network at the identified earlier checkpoint. In some implementations, the system 100 can determine whether to deploy the trained target neural network 110 for use in processing network inputs received from users based on the measure of bias 150. As an example, the system 100 can determine to deploy the trained target neural network 110 for use in processing network inputs if the measure of bias 150 does not meet the threshold measure of bias. Thus, in response to determining to deploy the trained target neural network, the system 100 can deploy the DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application trained target neural network 110 for use in processing network inputs received from users. In some implementations, the trained target neural network 110 is one of multiple trained target neural networks. For example, the multiple trained target neural networks can include trained target neural networks with different architectures or trained target neural networks of different sizes. Alternatively or in addition, the multiple trained target neural networks can have been trained using different training techniques, or trained on different training data. In these implementations, the system 100 can determine which of the trained target neural networks to deploy. For example, the system can determine a respective measure of bias for each of the trained target neural networks. The system can identify a trained target neural network that has a respective measure of bias that represents a lowest amount of bias among the respective measures of bias. The system can thus deploy the identified trained target neural network. In some implementations, the system 100 can receive a request to deploy the trained target neural network 110 from a requester. For example, the requester can include a user device, e.g., a computing device that receives user input from a user of the user device. The system 100 can determine whether to deploy the trained target neural network based on the measure of bias as described above. If the system 100 determines to deploy the trained target neural network 110, the system 100 can deploy the trained target neural network 110 for use in processing network inputs received from users. If the system 100 determines not to deploy the trained target neural network 110, the system 100 can automatically notify one or more monitors of the request of the determination not to deploy the trained target neural network 110. As an example, if the requester is a user device, a monitor of the request can include a user of the user device. To automatically notify the monitor, the system 100 can cause data representing the notification of the determination to be displayed to the monitor. For example, the system 100 can send an email to the monitor by providing data representing the email to the requester. As another example, the system 100 can send a push notification to the monitor by providing data representing the push notification to the requester. In some examples, in response to determining that the measure of bias 150 meets a threshold measure of bias, the system 100 can deploy the trained target neural network 110 with de-biasing prompts. For example, the trained target neural network 110 can DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application process a prompt that is derived from a network input received from a user and that includes an instruction to be unbiased. As another example, the system 100 can prepend learned embeddings to the prompt that are jointly trained with adversarial de-biasing and a contrastive loss. Further details are described in Berg et al., A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning. arXiv preprint arXiv:2203.11933, 2022. FIG. 3 is a flow diagram of an example process 300 for determining bias for a trained target neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system for determining bias for a trained target neural network, e.g., the system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300. The system obtains multiple initial network inputs (step 310). Each of the initial network inputs has been classified as belonging to a respective ground truth class, and each of the initial network inputs is associated with a corresponding feature value of a particular feature. The corresponding feature value is one of multiple feature values for the particular feature. In some examples, each of the initial network inputs includes a data item, such as text, video data, image data, or audio data. In some examples, each of the initial network inputs represents measurements captured by one or more sensors, such as images captured by camera sensors, audio captured by audio sensors, or observations captured by sensors that sense an environment being interacted with by an agent. The system processes each of the multiple initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input (step 320). Each respective predicted network output defines a respective predicted class for the initial network input. The trained target neural network can be represented by ^^ఏ∶ ^^ ^ ^^ with parameters ^^, where X is the set of inputs and Y is the set of predicted classes. Each input ^^ ∈ ^^ with predicted class ^^ ∈ ^^ is drawn from an underlying distribution^^^^^|^^, ^^^, where z is a discrete latent variable ^^ ∈ ^^ that represents a factor of variation,such as different feature values for a feature. z is also referred to as the bias variable and affects how well the model ^^ఏis able to predict y from x. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The system determines, for each class of the respective ground truth classes for the initial network inputs, a respective effect size (step 330). The respective effect size represents an association between the class and the particular feature. In some examples, the system determines a respective effect size for each class of the predicted classes. Determining a respective effect size for a particular class is described below with reference to FIG.4. The system determines a measure of bias of the trained target neural network (340). For example, the system can aggregate the respective effect sizes over each of the respective ground truth classes for the initial network inputs. As an example, the system can derive the measure of bias from each of the respective effect sizes for each of the respective ground truth classes and a mean of the respective effect sizes for each of the respective ground truth classes. For example, the system can compute the measure of bias based on respective differences between each of the respective effect sizes and a mean effect size for the respective effect sizes. As a particular example, for effect sizes ^ ^^^, ^^ଶ, … , ^^|^|^ with empirical mean ^̅^ , the system can determine the measure of bias as: That is, the system can compute a sum over each respective ground truth class of a cubed difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes, divided by a sum over each respective ground truth class of a squared difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes raised to a power of 3 / 2. In some examples, the system can derive the measure of bias from each of the respective effect sizes for each of the respective predicted classes and a mean of the respective effect sizes for each of the respective predicted classes. For example, the system can compute a sum over each respective predicted class of a cubed difference between the respective effect size for the respective predicted class and the mean of the respective effective sizes, divided by a sum over each respective predicted class of a squared difference between the respective effect size for the respective predicted class and the mean of the respective effective sizes raised to a power of 3 / 2. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The measure of bias presented above captures both how asymmetrical the distribution of estimated effect size values is, as well as the direction of the asymmetry. For example, the measure of bias indicates an overall weaker bias in case the distribution of effect size values per class is centered around zero, with infrequent higher values (as classes for which the model is strongly affected by bias are rare). The measure of bias also distinguishes models weakly affected by bias from models where for many classes the predictions exhibit high degrees of association with the feature (that is, the distribution of effect size values is long-tailed and skewed towards the left). In some examples, the system can provide data representing the measure of bias to a user, e.g., through a user interface. In some examples, the system can determine whether to further train the trained target neural network or whether to deploy the trained target neural network based on the measure of bias, as described above with reference to FIG.2. In some implementations, the system can determine a sufficiency of the distribution by conditioning on the predicted class and using the ground-truth class as the independent variable in the ^^ଶtest described below with reference to FIG. 4. In some implementations, the system can determine an independence fairness criterion such as Democratic Parity (DP) by using the predicted network outputs as the independent variable. In some implementations, the system can determine the measure of bias based on more than one respective predicted class for each initial network input. For example, the system can process each of the multiple initial network inputs using the trained target neural network to generate a respective predicted network output for each initial network input, where each respective predicted network output defines the top-k respective predicted classes for the initial network input. In some implementations, the system can determine the measure of bias based on logits or softmax scores. For example, the system can process each of the multiple initial network inputs using the trained target neural network. For each initial network input, the system can obtain the logits or softmax score for the respective predicted network output. For example, this specification describes determining a measure of bias by computing a chi-squared statistic. In some implementations, the system can determine a measure of bias using a statistical test such as the Mann-Whitney U test, based on logits or softmax scores for respective predicted network outputs. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application FIG.4 is a flow diagram of an example process 400 for determining an effect size for a class. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a system for determining bias for a trained target neural network, e.g., the system 100 of FIG.1, appropriately programmed in accordance with this specification, can perform the process 400. The system performs steps 410-440 as part of step 330 of FIG. 3. The system performs steps 410-440 for each particular class of the respective ground truth classes. In some implementations, the system performs steps 410-440 for each particular class of the predicted classes. The system identifies initial network inputs for which the respective ground truth class is the particular class (step 410). For example, the system identifies initial network inputs from the initial network inputs obtained in step 310 of FIG.3. The system identifies the predicted network outputs for the identified initial network inputs (step 420). For example, the system identifies predicted network outputs from the respective predicted network outputs generated in step 320 of FIG.3. For each class of the respective ground truth classes, the system determines, for each feature value, a respective count of identified initial network inputs that are associated with the feature value and for which the predicted network output defines the class (step 430). In some implementations where the system performs steps 410-440 for each particular class of the predicted classes, the system determines, for each class of the predicted classes, and for each feature value, the respective count of identified initial network inputs that are associated with the feature value and for which the predicted network output defines the class. The system determines the effect size for the particular class based on the respective counts of identified initial network inputs for the feature values and the respective predicted classes (step 440). The effect size v varies from 0 to 1, with 1 indicating that the predictions were extremely affected by changes in the feature values. As an example, the system can compute a square root of a chi-squared statistic for the respective counts for the feature values divided by a count of identified predicted network outputs multiplied by a number of degrees of freedom. For example, the system can determine the effect size v for a particular class as: DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application where N is the sample size, i.e., a count of identified predicted network outputs, and DF is the number of degrees of freedom. The number of degrees of freedom is based on a number of respective predicted classes or a number of feature values. For example, the number of degrees of freedom can be the smaller of the number of respective predicted classes and the number of feature values, subtracted by one. ^^ଶrepresents a test statistic from a Pearson’s chi-squared independence test. As an example, the chi-squared statistic is derived from a sum over each respective count for the feature values and the respective predicted classes of a squared difference between the respective count and a corresponding expected count divided by the corresponding expected count. For example, the system can determine ^^ଶas: where ^^ ൌ | ^̅^| ^ |Y| is the number of combinations of feature values and predicted classes , ^^^is the respective count for the kth combination of feature value and predicted class, and ^^^is the corresponding expected count. The corresponding expected count can be obtained from the multiple initial network inputs. In some examples, the system can maintain the respective counts in a contingency table of size ^^ ൌ | ^̅^| ^ | ^^| . For a given class ^^′, each entry of the contingency table corresponds to the frequency with which each predicted class was observed per feature value in the network outputs. In general, for a fixed value of ^^′ ∈ ^^, distributional bias accounts for systematic differences in the outcomes of ^^ఏ^ ^^^ across different feature values, i.e., when x is sampled from ^^^ ^^| ^^, ^^ ൌ ^^^ versus ^^^ ^^| ^^, ^^ ൌ ^^^ . Distributional bias can be defined as a comparison between families of distributions defined by ^^ఏ^ ^^^ when x is given by ^^^ ^^| ^^ ൌ ^^′, ^^ ൌ ^^^ versus when x is given by ^^^ ^^| ^^ ൌ ^^′, ^^ ൌ ^^^: ^^^ ^^^^ ^^| ^^; ^^^|| ^^^^ ^^| ^^; ^^^^ Where ^^^^ ^^| ^^; ^^^ and ^^^^ ^^| ^^; ^^^ denote the family of distributions obtained when the bias variable assumes each of its possible values. ^^^^ || ^^ represents an operator that accounts for a notion of similarity between the distributions. In examples where ^^ఏparameterizes a family of Gaussian distributions, ^^ can denote the KL- divergence between ^^^and ^^^. In examples where ^^ఏ^ ^^^ parameterizes families of DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application categorical distributions, such as in classification tasks, the comparison between ^^^and ^^^can be formulated as estimating the effect size of the association between the bias variable z and the observed model predictions ^^ᇱ~ ^^^ ^^| ^^, ^^^ . Thus the measure of similarity can be estimated using the effect size. FIG. 5 shows the performance of an example system for determining bias for a trained target neural network. In the example of FIG. 5, the trained target neural networks, “Model 1” (M1) and “Model 2” (M2) are image classification models that have been trained to predict occupation of a subject depicted in an image. In some examples, M1 and M2 can have been trained differently, or have different architectures. FIG. 5 shows a comparison between a “SKEWSIZE” measure of bias (corresponding to the measure of bias determined by the system for determining bias described in this specification) and several other measures for the two trained target neural networks. The other measures include model accuracy in each subgroup, worst group accuracy (minimum accuracy across subgroups), and GAP (the difference between worst subgroup accuracy and overall accuracy). FIG.5 shows that the other measures do not surface the bias of the trained target neural networks. The graphs 510a and 510b show the prediction counts of M1 and M2, respectively, over the predicted classes “writer,” “editor,” composer,” and “philosopher,” where the ground truth class for the initial network input was “writer.” For each predicted class, the graphs 510a and 510b show the prediction counts across two subgroups, or feature values, “male” and “female.” FIG. 5 shows that M2’s accuracy and worst group accuracy are lower than those of M1. Graph 510b shows a bias in M2’s predictions, in women being misclassified as editors, and men being misclassified as composers and philosophers. FIG. 5 shows that the measures of accuracy and worst group accuracy degrade for M2, but GAP does not. The graphs 520a and 520b show the prediction counts of M1 and M2, respectively, over the predicted classes “doctor,” “surgeon,” “nurse,” and “teacher,” where the ground truth class for the initial network input was “doctor.” FIG.5 shows that M2 and M1 have similar accuracies. However, the graphs 520a and 520b show a bias in M2’s predictions, with women being misclassified as nurses, and men being misclassified as surgeons. FIG. 5 shows that the measures of accuracy, worst group accuracy, and GAP do not capture this bias. The SKEWSIZE measure of bias captures this bias. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The graphs 530a and 530b show the prediction counts of M1 and M2, respectively, over the predicted classes “biologist,” “scientist,” teacher,” and “banker,” where the ground truth class for the initial network input was “biologist.” FIG.5 shows that M2 has a higher accuracy than M1. However, the graphs 530a and 530b show a bias in M2’s predictions, with women being misclassified as teachers and men being misclassified as scientists or bankers. FIG. 5 shows that the measures of accuracy, worst group accuracy, and GAP either improve for M2 or are the same as for M1. The SKEWSIZE measure of bias captures this bias. FIG.5 also shows that when aggregating across the different ground truth classes, the measures of accuracy, worst group accuracy, and GAP either improve in M2 relative to M1 or do not change. Thus the SKEWSIZE measure of bias captures the bias of M2 while the other measures do not. Examples of environments and observations for when the classification task is an agent control task now follow. In some implementations, the environment is a real-world environment, the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application signal; and / or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment. In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements, e.g., steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force / torque / acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation, e.g., steering, and movement, e.g., braking and / or acceleration of the vehicle. In some implementations the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example, the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world. In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material, e.g., to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g., robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g., via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot. The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example, the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines. As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g., minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process. The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment, e.g., between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine. The rewards or return may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g., a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use a resource the matric may comprise any metric of usage of the resource. In general observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and / or mechanical items of equipment. For example, a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g., DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate or flow / movement path of one or more materials; physical or chemical conditions, e.g., a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g., data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment. In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control, e.g., cooling equipment, or air flow control or air conditioning equipment. The task may comprise a task to control, e.g., minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g., environmental, control equipment. In general the actions may be any actions that have an effect on the observed state of the environment, e.g., actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g., actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application In general observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example, a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open. The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control, e.g., minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource. In some implementations the environment is the real-world environment of a power generation facility, e.g., a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g., to control the delivery of electrical power to a power distribution grid, e.g., to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements, e.g., to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g., an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated. The rewards or return may relate to a metric of performance of the task. For example in the case of a task to control the delivery of electrical power to the power distribution grid the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility. In the case of a task to maximize the delivery of electrical power to the power distribution grid the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility. In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example, a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid, e.g., from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid. As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals / intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that performs or controls the protein folding actions or chemical synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical / intermediates / precursors and / or may be derived from simulation. In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmachemical drug and the agent is a DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application computer system for determining elements of the pharmachemical drug and / or a synthetic pathway for the pharmachemical drug. The drug / synthesis may be designed based on a reward derived from a target for the drug, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the drug. In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources, e.g., on a mobile device and / or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources. As further example, the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users. In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location). As another example the environment may be an electrical, mechanical or electro- mechanical design environment, e.g., an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize the entity, i.e., observations of a mechanical shape or of an electrical, mechanical, or electro-mechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity, e.g., that modify one or more of the observations. The rewards or return may comprise one or more metric of performance of the design of the entity. For example rewards or return may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, e.g., in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus a design an entity may be optimized, e.g., by reinforcement learning, and then the optimized design output for DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application manufacturing the entity, e.g., as computer executable instructions; an entity with the optimized design may then be manufactured. As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment. The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real-world environment or real- world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. For example, the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment. Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, or both. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non‑transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially‑generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application‑specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand‑alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub‑programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read‑only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto‑optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer‑readable media suitable for storing computer program instructions and data include all forms of non‑volatile memory, media and memory devices, including by DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto‑optical disks; and CD‑ROM and DVD-ROM disks. To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back‑end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front‑end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back‑end, middleware, or front‑end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application CLAIMS 1. A method performed by one or more computers, the method comprising: obtaining a plurality of initial network inputs, wherein each of the initial network inputs has been classified as belonging to a respective ground truth class, and wherein each of the initial network inputs is associated with a corresponding feature value of a particular feature, wherein the corresponding feature value is one of a plurality of feature values for the particular feature; processing each of the plurality of initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input, the respective predicted network output defining a respective predicted class for the initial network input; determining, for each class of the respective ground truth classes for the initial network inputs, a respective effect size that represents an association between the class and the particular feature; and determining a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth classes for the initial network inputs.

2. The method of claim 1, wherein the trained target neural network processes network inputs that represent measurements captured by one or more sensors.

3. The method of claim 2, wherein the measurements comprise images captured by one or more camera sensors.

4. The method of claim 2 or claim 3, wherein the measurements comprise audio captured by one or more audio sensors.

5. The method of any one of claims 2-4, wherein the measurements comprise observations captured by one or more sensors that sense an environment being interacted with by an agent.

6. The method of any preceding claim, wherein the trained target neural network has been trained on a first subset of training data, and obtaining a plurality of initial network inputs that have been classified as belonging to a respective ground truth class comprises: obtaining the plurality of initial network inputs from a second subset of the training data.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 7. The method of any preceding claim, wherein obtaining a plurality of initial network inputs, wherein each of the initial network inputs has been classified as belonging to a respective ground truth class comprises: generating one or more input representations that describe a ground truth class of the respective ground truth classes and a corresponding feature value of the plurality of feature values; and for each of the one or more input representations, processing the input representation using a first generative neural network to generate one or more initial network inputs that are conditioned on the input representation.

8. The method of claim 7, wherein each input representation comprises a respective text prompt that describes the ground truth class and the corresponding feature value and wherein the first generative neural network has been trained to generate network outputs that are described by input text prompts.

9. The method of claim 8, wherein each input representation is generated using a language model that is configured to generate a text sequence that comprises a text prompt that describes a ground truth class and a corresponding feature value, a query about the ground truth class, and an answer that corresponds to the ground truth class, and wherein the input representation includes the text prompt.

10. The method of claim 9, wherein each initial network input comprises the network output generated using the first generative neural network, and the query of the text sequence generated using the language model.

11. The method of any one of claims 8-10, wherein the first generative neural network has been trained to generate network outputs that comprise images that are described by input text prompts.

12. The method of claim 10, wherein each initial network input comprises an image, and the query for each initial network input comprises a text question about the image for each initial network input.

13. The method of any preceding claim, wherein the initial network input comprises an image, and the trained target neural network has been trained to generate a predicted network output that comprises a predicted class for the image.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 14. The method of any preceding claim, wherein the trained target neural network has been trained to generate a predicted network output that is a text description of a predicted class for the initial network input.

15. The method of claim 14, wherein processing each of the plurality of initial network inputs using a trained target neural network to generate a respective predicted network output for each initial network input comprises: determining that the text description in the network output semantically matches a description of one of the respective ground truth classes for the initial network inputs; and setting the predicted class for the initial network input to be the ground truth class.

16. The method of any preceding claim, further comprising: determining that the measure of bias is over a threshold measure of bias; and in response to determining that the measure of bias meets a threshold measure of bias, further training the trained target neural network.

17. The method of any preceding claim, further comprising determining that the measure of bias is over a threshold measure of bias during training of the trained target neural network.

18. The method of claim 17, further comprising: in response to determining that the measure of bias meets a threshold measure of bias during training of the trained target neural network, halting training of the trained target neural network.

19. The method of claim 18, further comprising: determining to deploy a previous checkpoint of the trained target neural network, wherein the previous checkpoint of the trained target neural has a measure of bias that does not meet the threshold measure of bias.

20. The method of any preceding claim, further comprising determining that the measure of bias for the trained target neural network represents a higher amount of bias than one or more respective measures of bias for one or more earlier checkpoints of the trained target neural network.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 21. The method of claim 20, further comprising, in response to determining that the measure of bias for the trained target neural network represents a higher amount of bias than one or more of the respective measures of bias for one or more earlier checkpoints of the trained target neural network, deploying a checkpoint of the one or more earlier checkpoints of the trained target neural network.

22. The method of claim 16, wherein further training the trained target neural network comprises: obtaining training data that the trained target neural network was trained on; modifying the training data with respect to the particular feature; and further training the trained target neural network on the modified training data.

23. The method of claim 22, wherein modifying the training data with respect to the particular feature comprises obtaining additional training data.

24. The method of claim 22, wherein modifying the training data with respect to the particular feature comprises generating training data using one or more neural networks.

25. The method of claim 24, wherein generating training data using one or more neural networks comprises de-biasing prompts to the one or more neural networks.

26. The method of claim 16, wherein further training the trained target neural network comprises using guidance to train the trained target neural network.

27. The method of claim 16, wherein further training the trained target neural network comprises using an adversarial loss to train the trained target neural network.

28. The method of claim 16, wherein further training the trained target neural network comprises fine-tuning the trained target neural network.

29. The method of claim 16, wherein further training the trained target neural network comprises unlearning a subset of training data that the trained target neural network was trained on.

30. The method of any preceding claim, further comprising: determining whether to deploy the trained target neural network for use in processing network inputs received from users based on the measure of bias.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 31. The method of claim 30, further comprising: in response to determining to deploy the trained target neural network, deploying the trained neural network for use in processing network inputs received from users.

32. The method of claim 30 or 31, further comprising: receiving a request to deploy the trained target neural network from a requester; and in response to determining not to deploy the trained target neural network, automatically notifying one or more monitors of the request of the determination not to deploy the trained target neural network.

33. The method of claim 32, wherein automatically notifying the one or more monitors of the request of the determination not to deploy the trained target neural network comprises sending an email to the one or more monitors.

34. The method of claim 32 or 33, wherein automatically notifying the one or more monitors of the request of the determination not to deploy the trained target neural network comprises sending a push notification to the one or more monitors.

35. The method of any preceding claim, further comprising: receiving, from a user, a user input that represents the particular feature.

36. The method of any preceding claim, further comprising: providing the measure of bias to a user.

37. The method of any preceding claim, wherein the trained target neural network is one of a plurality of trained target neural networks, and wherein the method further comprises: determining a respective measure of bias for each of the trained target neural networks; and identifying a trained target neural network that has a respective measure of bias that represents a lowest amount of bias among the respective measures of bias.

38. The method of claim 37, further comprising deploying the identified trained target neural network.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 39. The method of any preceding claim, wherein determining, for each class of the respective ground truth classes for the initial network inputs, a respective effect size comprises: for each particular class of the respective ground truth classes: identifying initial network inputs for which the respective ground truth class is the particular class; identifying the predicted network outputs for the identified initial network inputs; for each class of the respective ground truth classes, determining, for each feature value, a respective count of identified initial network inputs that are associated with the feature value and for which the predicted network output defines the class; and determining the effect size for the particular class based on the respective counts of identified initial network inputs for the feature values and the respective predicted classes.

40. The method of claim 39, wherein determining the effect size for the particular class comprises computing a square root of a chi-squared statistic for the respective counts for the feature values divided by a count of identified predicted network outputs multiplied by a number of degrees of freedom, wherein the number of degrees of freedom is based on a number of respective predicted classes or a number of feature values.

41. The method of claim 40, wherein the chi-squared statistic is derived from a sum over each respective count for the feature values and the respective predicted classes of a squared difference between the respective count and a corresponding expected count divided by the corresponding expected count.

42. The method of claim 41, wherein the corresponding expected count is obtained from the plurality of initial network inputs.

43. The method of any preceding claim, wherein determining a measure of bias of the trained target neural network with respect to the particular feature by aggregating the respective effect sizes over each of the respective ground truth classes for the initial network inputs comprises deriving the measure of bias from each of the respective effect sizes for each of the respective ground truth classes and a mean of the respective effect sizes for each of the respective ground truth classes.DeepMind Technologies Limited F&R Ref.: 45288-0383WO1 PCT Application 44. The method of claim 43, wherein deriving the measure of bias from each of the respective effect sizes for each of the respective ground truth classes and a mean of the respective effect sizes for each of the respective ground truth classes comprises computing a sum over each respective ground truth class of a cubed difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes, divided by a sum over each respective ground truth class of a squared difference between the respective effect size for the respective ground truth class and the mean of the respective effective sizes raised to a power of 3 / 2.

45. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of claims 1-44.

46. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-44.

Images