Neural network with improved performance, using automatically uncovered failure cases

Abstract

To provide a method, system and apparatus, including a computer program encoded on a computer storage medium, for adjusting a target neural network using automatically generated test cases before deployment of the target neural network in a deployment environment.SOLUTION: One of methods may include the steps of: generating a plurality of test inputs by using a test case generation neural network; processing the plurality of test inputs using a target neural network to generate one or more test outputs for each test input; and identifying, from the one or more test outputs generated by the target neural network for each test input, failing test inputs that result in generation of test outputs by the target neural network that fail one or more criteria.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 303,958, filed January 27, 2022. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference.

[0002] This specification relates to processing text using neural networks. [Background technology]

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e. the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current value of its respective set of weights. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rae et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher,” arXiv:2112.11446 [Non-Patent Document 2] Xu et al., “Bot adversarial dialogue for safe conversational agents,” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968, Association for Computational Linguistics. Summary of the Invention [Means for solving the problem]

[0005] Described herein is a system implemented as a computer program on one or more computers at one or more locations that can automatically generate test cases for a target neural network that is configured through training to perform a machine learning task on a target network input. The target neural network can be any of the various types and versions of neural networks that have been generated for machine learning tasks. These neural networks can be based on various architectures, technologies, languages, vocabularies, etc., and can be trained using various training data.

[0006] After training, and prior to deployment in a production environment, the target neural network may need to be validated and evaluated for suitability for deployment. For example, the target neural network may need to be evaluated to determine whether the network outputs generated by the trained target neural network fail to meet one or more criteria, and if so, how frequently. Unlike traditional scenarios that rely on human annotators to hand-write test cases for pre-deployment validation and evaluation, the present system implements and uses a test case generation neural network to automatically generate test inputs.

[0007] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0008] Some of the described techniques can be used to automatically generate any number of test cases for a text-generating neural network to identify potentially harmful or somehow undesirable behavior of the text-generating neural network, such as outputting text that includes offensive or sensitive content. Some of the described techniques can then be used to analyze "failing" test cases and find common failure patterns. Appropriate measures can then be taken to improve the performance of the text-generating neural network, for example to protect it from generating offensive or sensitive content.

[0009] Adverse behaviors of neural networks have traditionally been identified by human annotators handwriting test cases prior to deploying the neural network. Human annotation is expensive, for example, in terms of costs associated with the human intelligence required to write the test cases and the difficulty of scaling to the large number of test cases required for complex target neural networks. Furthermore, both the number and diversity of test cases are limited by human annotation. In contrast, by using a neural network (sometimes instantiated as another identical instance of a text generating neural network) along with prompt design techniques to automatically generate test cases, the techniques described herein can generate a significantly larger number of test cases with significantly increased diversity of the text contained in those test cases, while minimizing human intervention.

[0010] The described technique thus facilitates proactive identification and correction of potentially harmful network behaviors that were previously impossible and difficult, if not impossible, to discover prior to deployment. The described technique facilitates several technical improvements. For example, the computational costs required to train a text-generating neural network and prepare it for deployment may be reduced. As another example, the range of technical use cases for text-generating neural networks may be expanded. Thus, text-generating neural networks may become more suitable for deployment in production environments, such as within educational or medical organizations, where any harmful behavior of the deployed network may have significant consequences.

[0011] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0012] [Figure 1]1 illustrates an example of a pre-deployment evaluation system and an example of a production environment. [Diagram 2] FIG. 1 illustrates an evaluation of the performance of a target neural network against one or more criteria. [Diagram 3] 1 is a flow diagram of an example process for identifying failing test inputs. [Figure 4] 1 is a flow diagram of an exemplary process for generating test inputs by using a test case generation neural network. [Diagram 5] 11 is a flow diagram of an example process for identifying additional failing test inputs. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Like reference numbers and designations in the various drawings indicate like elements.

[0014] Described herein is a system, implemented as a computer program on one or more computers at one or more locations, that can automatically generate test cases for a target neural network configured to perform a machine learning task on a target network input.

[0015] 1 illustrates an example pre-deployment assessment system 100 and an example production environment 140. Pre-deployment assessment system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below are implemented.

[0016] Pre-deployment assessment system 100 includes target neural network 120. Target neural network 120 is an instance of a neural network that has been trained to perform a machine learning task, i.e., to process network inputs to generate network outputs for the machine learning task.

[0017] In general, the machine learning task that the target neural network 120 is configured to perform can be any of a variety of natural language modeling tasks. Some examples of natural language modeling tasks that the target neural network 120 can be configured to perform are described below.

[0018] As an example, the task may be a neural machine translation task. For example, if the input to the neural network is a sequence of text, e.g., a sequence of words, phrases, characters, or word fragments in one language, the output generated by the neural network may be a translation of the sequence of text into another language, i.e., a sequence of text in another language that is a translation of the input sequence of text. As a particular example, the task may be a multilingual machine translation task, where a single neural network is configured to translate between multiple different source and target language pairs. In this example, the source language text may be augmented with an identifier indicating the target language into which the neural network should translate the source language text.

[0019] As another example, the task may be a natural language processing or understanding task, such as an entailment task, a paraphrase task, a text similarity task, a sentiment task, a sentence completion task, a grammar task, etc., that operates on a sequence of text in some natural language.

[0020] As another example, the task may be a text-to-speech task where the input is text in a natural language or features of text in a natural language, and the network output is a spectrogram or other data defining the audio of the text spoken in the natural language.

[0021] As another example, the task may be a text generation task where the input is a sequence of text and the output is another sequence of text, e.g., a completion of the input sequence of text, a response to a question posed in the input sequence, or a sequence of text on a topic specified by the first sequence of text.

[0022] As another example, the task may be a digital assistant task where the input is a user's request for assistance and the output includes information related to the user request. In implementations, this may include generative (large-scale) language models, particularly for dialogue, e.g., conversational agents such as Gopher. For example, the output may include instructions for a sequence of actions to be performed by the user, e.g., steps or subtasks of an overall task to achieve a technical effect and / or solve a technical problem in a real-world scenario. The instructions may be generated, for example, in the form of natural language (transmitted as voice and / or text on a screen).

[0023] In some cases, the natural language modeling task is a combination of multiple individual natural language modeling tasks, i.e., the system is configured to perform multiple different individual natural language modeling tasks, e.g., two or more of the natural language modeling tasks listed above. For example, the system can be configured to perform multiple individual natural language understanding tasks with a network input that includes identifiers of the individual natural language understanding tasks performed in the network input.

[0024] The target neural network 120 can have any of a variety of suitable network architectures that enable it to perform these natural language modeling tasks. For example, the target neural network 120 can be configured as a recurrent neural network (e.g., a neural network including one or more long short-term memory (LSTM) layers), an attention-based neural network (e.g., a neural network including one or more attention layers), or another generative neural network. By way of example and not limitation, some exemplary architectures of the target neural network 120, as well as related techniques for pre-training the target neural network 120 using unlabeled text training data, are described in more detail in "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" by Rae et al., arXiv:2112.11446, the entire contents of which are incorporated herein by reference.

[0025] Training of the target neural network 120 may be performed locally at the pre-deployment assessment system 100 or may be performed at a separate cloud-based training system. For example, a training system, local or remote to the pre-deployment assessment system 100, may train the target neural network 120 at a first natural language modeling task using first unlabeled text training data to determine pre-trained values ​​for the parameters of the target neural network 120 (hereinafter referred to as "target network parameters"). Optionally, the training system may further fine-tune the pre-trained target neural network 120 for one or more specific downstream tasks, for example, using labeled text training data available for the specific downstream task. The training system may train the target neural network 120 until convergence of an unsupervised loss function suitable for the first natural language modeling task or until a fixed number of training steps have been performed.

[0026] In particular, after training, the pre-deployment assessment system 100 utilizes the test case generation neural network 110 to evaluate the performance of the target neural network 120 for its suitability for deployment in a production environment 140. Like the target neural network 120, the test case generation neural network 110 is an instance of a neural network that has been trained to perform a machine learning task.

[0027] For example, the training system may train the test case generating neural network 110 on a second natural language modeling task using second unlabeled text training data to determine pre-training values ​​for parameters (hereinafter referred to as "test case network parameters") of the test case generating neural network 110. The second natural language modeling task (and / or the second unlabeled text training data) may be the same as or different from the first natural language modeling task (and / or the first unlabeled text training data) on which the target neural network 120 was trained.

[0028] The test case generating neural network 110 can have any of a variety of suitable network architectures, similar to the network architecture of the target neural network 120. Indeed, in some implementations, the test case generating neural network 110 and the target neural network 120 can have the same network architecture, e.g., both can be configured as attention-based generative neural networks including one or more attention layers. As used herein, an attention layer is a neural network layer that includes an attention mechanism, e.g., a multi-headed self-attention mechanism. Furthermore, in some of these implementations, the networks 110 and 120 can both be instantiated with a common set of network parameter values, e.g., both can be instantiated with the same pre-trained parameter values ​​determined by training on the same natural language modeling task using the same set of unlabeled text training data.

[0029] At a high level, the pre-deployment assessment system 100 uses a test case generation neural network 110 to automatically generate a number of test inputs 112, which are then processed by a target neural network 120 in accordance with the target network parameters to generate test outputs 122. The pre-deployment assessment system 100 then evaluates these test outputs 122 by evaluating them against one or more criteria 131 to determine whether the target neural network 120 is suitable for deployment in a production environment 140.

[0030] In some implementations, the pre-deployment evaluation system 100 can output information about the evaluation results to a user indicating whether the target neural network 120 is suitable for deployment. In some implementations, if the target neural network 120 does not meet one or more of the criteria during the pre-deployment evaluation, the pre-deployment evaluation system 100 can adjust the network accordingly to improve the fitness of the network with respect to the one or more criteria, and if the target neural network 120 (or an adjusted target neural network obtained by using the techniques broadly described herein to adjust the target neural network 120) passes all the criteria, the pre-deployment evaluation system 100 can provide data to the production environment 140 specifying the target neural network 120 (or the adjusted target neural network) to be used to perform inference on a machine learning task, i.e., to generate online network outputs of the machine learning task for online network inputs, so that the target neural network 120 (or the adjusted target neural network) can be deployed to the production environment 140.

[0031] As used herein, a production environment refers to an environment in which a neural network may be applied to real-time user-input data, rather than training or test data for which the expected outcome is known. Input data in a production environment is sometimes referred to as online data, as opposed to offline training or test data.

[0032] The production environment 140 can be implemented on one or more computing devices. For example, the production environment 140 can deploy the target neural network 120 (or the tuned target neural network) in a data center that includes hundreds of computers for processing input received from remote users, or the production environment 140 can provide trained parameter values ​​of the target neural network 120 to edge devices, such as, for example, mobile phones, smart personal assistant devices, smart watches, smart displays, or other IoT devices, via a wired or wireless network connection to enable the target neural network 120 to be used to perform machine learning tasks on the edge devices.

[0033] The pre-deployment evaluation system 100 implements appropriate prompting techniques to effectively guide the process of generating test inputs 112 by using the test case generation neural network 110. Incorporating prompting techniques into the test case generation process can be advantageous for many reasons: first, the prompts improve the control of the evaluation process to find various failing test cases more quickly; second, the prompts provide guidance to the test case generation neural network 110 to generate specific types of test inputs, including test inputs with text that rarely occurs in the training data, thus facilitating the identification and analysis of specific failing patterns of the target neural network 120.

[0034] Thus, the test case generation neural network 110 is configured to receive test case network inputs 102 that are or include natural language prompts, and to process the test case network inputs 102 in accordance with the test case network parameters to generate test case network outputs from which test cases can be generated. The test cases include text inputs 112 that can be provided for processing by the target neural network 120. The natural language prompts are used to guide the test case generation neural network 110 to generate test cases of various levels of diversity and complexity. Generating natural language prompts as test case network inputs and generating text inputs from the test case network outputs are discussed in more detail below with reference to FIGS. 3-4.

[0035] Each test input 112 may include text. As used herein, text may refer to natural language text including one or more text elements (e.g., one or more words in a sentence) in one or more natural languages ​​including English, Chinese, Japanese, and Korean, to name just a few. Text may also refer to computer-readable text including one or more computer code elements in one or more computer programming languages ​​such as Python, C++, C#, Java, Ruby, PHP, etc.

[0036] The test inputs 112 generated in this manner are relatively inexpensive (e.g., in terms of the cost associated with the human intelligence required to create the test cases, the cost associated with the additional computational resources required to implement and train another test case generation model, or both) and cover a wide range of topics with different levels of difficulty to test, facilitating the discovery of any potentially harmful behaviors of the trained target neural network that may be different or impossible to discover manually. Moreover, unlike other conventional systems that attempt to evaluate a target neural network by brute force, e.g., by submitting gibberish or other incomprehensible content, the test inputs generated in this manner contain ostensibly reasonable text in natural language, thus mimicking text that may be submitted by users when deployed in a production environment.

[0037] The pre-deployment assessment system 100 provides test inputs 112 to a target neural network 120 and uses the target neural network 120 to process each test input 112 according to the target network parameters to generate one or more test outputs 122 for the test inputs 112. Each test output 122 may similarly include text. The test outputs 122 may be evaluated against any of a variety of criteria 131 maintained by the system 100 corresponding to the production environment 140. Data defining the criteria 131 may be uploaded by a user of the system 100 or may be determined by the system 100 itself based on known requirements of the production environment 140. The criteria 131 may be stored in a data store in association with different production environments. The criteria 131 may be periodically revised and updated as necessary.

[0038] For example, one or more criteria 131 may specify that text included in the test output should not include certain content or information. For example, the criteria may specify that the test output should not include one or more of offensive content, false information, or confidential or personal information, such as personal contact information. As another example, the criteria may specify that the test output should not cause unwanted data leakage. For example, in a federated learning situation where various organizations train a network together by using each organization's private local data, text from any private training data or other protectable data (e.g., copyrighted data) used to pre-train the target neural network 120 should not be included in the test output. In this example, data security of the target neural network may be improved because text from private training data or other protectable data is not included in the online network output sent to the user.

[0039] To automatically determine whether these test outputs 122 fail to meet one or more criteria, the pre-deployment assessment system 100 may also include a text-based classifier engine 130 that processes each test output 122 to determine whether the text output 122 fails to meet one or more criteria. In some implementations, the text-based classifier engine 130 may implement a text classifier machine learning model, such as a neural network, a logistic regression model, a support vector machine (SVM), or a decision tree or random forest model, or another black-box text classifier configured to process the test output 122 to generate as a classifier output 132 a predicted likelihood that the text output 122 fails to meet the one or more criteria.

[0040] By way of example and not limitation, several exemplary architectures of attention-based classification neural networks, and associated techniques for training such networks, are described in detail by Xu et al., “Bot adversarial dialogue for safe conversational agents,” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2950-2968, Association for Computational Linguistics, the entire contents of which are incorporated herein by reference.

[0041] In some implementations, the text-based classifier engine 130 may implement a deterministic text-based classification algorithm that generates a binary classifier output 132 indicating whether the text output 122 fails one or more criteria. Such deterministic algorithms may typically be defined using high-level computer programming code, such as Python or C++ code.

[0042] For example, if there is a single criterion, the classifier engine 130 can generate a probability score indicating that the test output 122 does not meet the criterion based on the output and, optionally, other information, such as the test input 112 or other context from the conversation history with the target neural network 120. As another example, if there are multiple criteria, the classifier engine 130 can include separate classifiers for different criteria, and the classifier engine 130 can generate a fail indication if the test output 122 does not meet any of the criteria. As yet another example, the classifier engine 130 can include a single classifier that makes a classification prediction for all of the criteria 131.

[0043] 2 is a diagram illustrating an evaluation of the performance of a target neural network against one or more criteria. In the example of FIG. 2, test outputs generated by the target neural network that include one or more of offensive or biased content, protectable information processed during training, or personal contact information do not pass one or more criteria.

[0044] Some implementations of the pre-deployment assessment system 100 can use the identified failing test inputs, i.e., test inputs 112 that produce test outputs 122 for which the target neural network 120 does not meet one or more criteria, to tune the target neural network 120 such that the tuned instance of the target neural network 120 produces test outputs that are less likely to fail one or more criteria, thereby improving the suitability of the tuned target neural network 120 for deployment in a production environment 140.

[0045] In some of these implementations, the pre-deployment assessment system 100 can tune the target neural network 120 based on analyzing the identified failing test inputs and applying natural language processing techniques to determine specific text content that, if included in the test input, is likely to result in the target neural network 120 generating a test output that does not meet one or more criteria. For example, a text clustering algorithm can be applied to group various text segments occurring in the failing test inputs into multiple text clusters. The text segments can be words or phrases that include multiple words or other terms from a vocabulary. From these multiple text clusters, one or more specific text segments that occur most frequently, or, stated differently, that have the highest probability of resulting in the generation of a test output that does not meet one or more criteria, can be identified.

[0046] The pre-deployment assessment system 100 can then adjust the target neural network 120 based on taking appropriate measures to address such content. For example, training examples from the unlabeled text training data that include particular text content can be removed and therefore not processed by the target neural network 120 during training, and are also not used in the calculation of losses that are used to update the values ​​of the target network parameters. As another example, training examples that include particular text content can be adjusted or modified before being processed by the target neural network 120, for example, by removing particular text segments from the training examples or adding additional text segments to the training examples.

[0047] Additionally or alternatively, some implementations of pre-deployment assessment system 100 can use the identified failing test inputs to improve the fitness of target neural network 120 after deployment in production environment 140. For example, system 100 can make improvements by sending instructions to one or more computing devices implementing production environment 140 to configure target neural network 120 to withhold providing any online network outputs or to generate default outputs in response to user-submitted online network inputs that include such content.

[0048] 3 is a flow diagram of an exemplary process 300 for identifying failing test inputs. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a system, such as pre-deployment assessment system 100 of FIG. 1, suitably programmed in accordance with this specification, may perform process 300.

[0049] The system generates 302 a plurality of test inputs by using a test case generating neural network having a plurality of test case network parameters. Each test input can include text. In some implementations, the test case generating neural network can be a pre-trained generative neural network model.

[0050] For example, the test case generation neural network may be an autoregressive generative model pre-trained on a first natural language modeling task using the first unlabeled text training data. In this example, the test case network output based on which the one or more test inputs are generated may include, at each of a plurality of test case network output time steps, a test case network score for each of the text tokens (or, for short, "tokens") in the vocabulary of the candidate tokens, where each token may represent one or more of a character, a word fragment, a word, a punctuation mark, etc. The test case generation neural network may be configured to process the test case network inputs according to a plurality of test case network parameters to autoregressively generate the test case network scores, e.g., by generating, at each of a plurality of test case network output time steps, a score distribution over the vocabulary of the candidate tokens that includes a test case network score for each of the candidate tokens adjusted based on the score distributions at one or more previous test case network output time steps. In particular, the test case network inputs may include natural language prompts. Generation of such natural language prompts is described below with reference to FIG. 4.

[0051] The system processes the multiple test inputs to generate one or more test outputs for each test input by using a target neural network having multiple target network parameters (step 304). Each test output may also include text. In some implementations, the target neural network may also be a pre-trained generative neural network model. Furthermore, in some implementations, the test case generative neural network and the target neural network may have the same network architecture and may be trained on the same task by using the same training data.

[0052] For example, the target neural network may be an autoregressive generative model pre-trained on a second natural language modeling task using second unlabeled text training data, which may be the same as or different from the first natural language modeling task (and / or the first unlabeled text training data). In this example, the target neural network may be configured to process each test input according to a plurality of target network parameters to generate a target network output that specifies a test output corresponding to the test input.

[0053] Similar to the test case generation neural network, the target neural network can do this by generating a respective target network score for each candidate token in the vocabulary of candidate tokens at each of a number of target network output time steps, and then using that score to select a token for the time step. Test outputs can be generated by the system based on the concatenation of these selected tokens.

[0054] The system identifies failing test inputs that result in the production of test outputs by the target neural network that do not meet one or more criteria (step 306). Specifically, the system can evaluate the test outputs against any of a variety of criteria, e.g., by utilizing one or more text-based classifiers to determine which particular test outputs fail to meet the criteria, and accordingly identifies test inputs that caused the target neural network to produce particular test outputs.

[0055] 4 is a flow diagram of an exemplary process 400 for generating test inputs by using a test case generation neural network. For convenience, the process 400 is described as being performed by a system of one or more computers located at one or more locations. For example, a system, such as the pre-deployment assessment system 100 of FIG. 1, suitably programmed in accordance with this specification, may perform the process 400.

[0056] The system generates natural language prompts as test case network inputs (step 402). The natural language prompts may include text, numbers, punctuation, or a combination thereof, and possibly other information. For example, the natural language prompts may be multiple word sentences describing a given topic that are generated based on a test case template for the given topic. The test case template may define fixed text segments that are included in the multiple natural language prompts and variable text segments that vary among the multiple natural language prompts.

[0057] In some implementations, the system can generate natural language prompts by iterating through a list of given topics known to be associated with the production environment. In some implementations, the system can generate natural language prompts based on previously generated natural language prompts, for example, by reusing a particular text segment (or content similar to a particular text segment) from a particular previously generated natural language prompt.

[0058] The system processes the test case network inputs according to the multiple test case network parameters using the test case generation neural network to autoregressively generate test case network outputs over multiple test case network output time steps (step 404). At each of the multiple test case network output time steps, the test case network output includes a respective test case network score for each candidate token in the vocabulary of candidate tokens.

[0059] The system generates test inputs based on the sampled tokens according to the test case network scores included in the test case network output (step 406). Specifically, the system can repeatedly sample sampled tokens from the vocabulary of candidate tokens according to the test case network scores corresponding to each of the multiple test case network output time steps. Thanks to sampling, i.e., by sampling different tokens to include them in different test inputs, the system can generate multiple text inputs from a single test case network output.

[0060] In some implementations of the system, more advanced token sampling techniques can be implemented to improve the quality of the test inputs. As an example, at each of some of the multiple test case network output time steps, the system can sample sampled tokens from a subset of the candidate tokens of the vocabulary (rather than from the entire vocabulary) as tokens to be included in the test input. The subset can include, for example, candidate tokens whose test case network scores are generated by the test case generation neural network with test case network scores greater than a certain threshold, e.g., test case network scores that are in the top 90% or top 95% of all scores. As another example, the system can repeatedly sample different tokens to be included in different test inputs until a predetermined number of different test inputs are generated, each of which meets a validity criterion. For example, the validity criterion can specify that each test input must contain a required termination string or otherwise meet a termination criterion.

[0061] In general, process 300 may be repeated as necessary to evaluate the performance of a target neural network that has undergone multiple training steps. For example, process 300 may be repeated hourly, daily, or weekly during the training process of the target neural network. Process 300 may also be triggered before the target neural network is deployed in a production environment.

[0062] After one or more iterations of process 300, some implementations of the system may adjust values ​​of test case network parameters of the test case generation neural network based on knowledge gained from previous iterations of process 300 to update the way in which test cases for the target neural network are generated by using the test case generation neural network, thereby improving the diversity, complexity, or both of the test cases generated in the next iteration of process 300.

[0063] As an example, a supervised learning (SL) technique can be used. The supervised learning technique can evaluate a learning target that depends on the total number of failing test inputs, e.g., log-likelihood, generated in one or more previous iterations of the process 300 by using the test case generation neural network. The system can use such a supervised learning technique to fine-tune the pre-trained values ​​of multiple test case network parameters, so that the test case generation neural network is modified in a way that produces a test case network output in which more failing test inputs are generated in the next iteration of the process 300.

[0064] As another example, reinforcement learning (RL) techniques can be used. The system can use such reinforcement learning to (i) further adjust the fine-tuned values ​​or (ii) fine-tune the pre-trained values ​​of the multiple test case network parameters, such that the test case generation neural network is modified in a way that produces test case network outputs that are more likely to produce failing test inputs in the next iteration of process 300. For example, the reinforcement learning technique can be an actor-critic technique that optimizes a reinforcement learning loss that includes a loss term that depends on the diversity of the test case network's outputs, e.g., a Kullback-Leibler (KL) information term.

[0065] As yet another example, the sub-sampling technique described in FIG. 5 can be used.

[0066] 5 is a flow diagram of an exemplary process 500 for identifying further failing test inputs using subsampling techniques. For convenience, process 500 is described as being performed by a system of one or more computers located at one or more locations. For example, a system, such as pre-deployment assessment system 100 of FIG. 1, suitably programmed in accordance with this specification, may perform process 500.

[0067] For example, the system may perform process 500 after one or more iterations of process 300 and after training the test case generation neural network using either or both of SL or RL techniques.

[0068] The system generates a number of additional test inputs, step 502. The system repeatedly performs sub-steps 504-510 of step 502 to generate each additional test input, as discussed below.

[0069] The system samples one or more test inputs from the plurality of test inputs generated by using the test case generation neural network (step 504). In some implementations, the system may sample one or more test inputs generated from one or more previous iterations of the process 300 with uniform randomness. In some other implementations, the system may sample one or more test inputs from a distribution that gives a higher probability of sampling a failing test input than the probability of sampling other test inputs from the plurality of non-failing test inputs. In other words, the system performs sampling in such a way that a failing test input is more likely to be sampled.

[0070] The system generates another test case network input that includes the one or more sampled test inputs, step 506. In some implementations, the system can use text included in the one or more sampled test inputs as the another test case network input, while in other implementations, the system can prepend or append text included in the one or more sampled test inputs to a natural language prompt to form the another test case network input.

[0071] The system processes the alternative test case network input according to the plurality of test case network parameters using the test case generation neural network to generate an alternative test case network output (step 508).

[0072] The system generates additional test inputs based on the sampling tokens according to the test case network scores contained in the other test case network outputs (step 510). Steps 508 and 510 are similar to steps 404 and 406 in FIG. 4 (but are performed based on test case network inputs generated in a different manner).

[0073] The system processes multiple additional test inputs using the target neural network to further identify failing test inputs, step 512. Step 512 is similar to step 306 in FIG.

[0074] The term "configured" is used herein in connection with systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. One or more computer programs configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0075] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded in an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device for execution by a data processing apparatus.

[0076] The term "data processing apparatus" refers to data processing hardware and encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or even include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an environment for the execution of computer programs, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0077] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, such as compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may correspond to a file in a file system, but this is not necessarily the case. A program may be stored in part of a file that holds other programs or data, for example in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, for example in files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to run on one computer, or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0078] The term "database" is used broadly herein to refer to any collection of data, which need not be structured in any particular way, or even at all, and which may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in different ways.

[0079] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and implemented on the same computer.

[0080] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data to generate output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as, for example, an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0081] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. In general, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be assisted by or incorporated in special-purpose logic circuitry. In general, a computer includes or is operatively coupled to receive or transfer data from one or more mass storage devices for storing data, e.g., magnetic, magneto-optical, or optical disks. However, a computer need not be equipped with such devices. Furthermore, a computer may be incorporated in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0082] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0083] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and a pointing device, such as, for example, a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as, for example, a smartphone implementing a messaging application, and receiving a response message from the user in return.

[0084] A data processing device for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling typical, computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0085] The machine learning model can be implemented and deployed using a machine learning framework, for example the TensorFlow framework or the JAX framework.

[0086] An embodiment of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0087] A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs implemented on the respective computers and having a client-server relationship to each other. In some embodiments, the server sends data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., the result of a user interaction, can be received from the device at the server.

[0088] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, even if features are described above as acting in a specific combination and initially claimed as such, one or more features from the claimed combination may in some cases be carved out of the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0089] Similarly, although operations are shown in the figures and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0090] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0091] 100 Pre-Deployment Evaluation System 102 Test Case Network Input 110 Test Case Generation Neural Network 112 Test Input 120 Target Neural Network 122 Test Output 130 Text-based Classifier Engine 131 Standards 132 Classifier output 140 Production Environment 300 processes 400 processes 500 processes

Claims

1. 1. A computer-implemented method comprising: maintaining a plurality of segments through a production environment in which a target neural network is deployed, the plurality of segments comprising: generating a plurality of test inputs by using a test case generation neural network; processing the plurality of test inputs using the target neural network to generate one or more outputs for each test input; identifying one or more failing test inputs that result in the target neural network producing a test output that does not meet one or more criteria based on the one or more test outputs produced by the target neural network for each test input; retaining the identified segments based on the one or more failed test inputs; receiving, by the production environment, online network input from a user; determining, by the production environment, that the online network input includes at least one of the plurality of segments; in response, generating, by the production environment, default outputs in response to the online network inputs without processing the online network inputs using the target neural network to generate online network outputs; A method comprising:

2. The method of claim 1 , wherein the production environment receives the plurality of segments from a pre-deployment environment system.

3. identifying the one or more failing test inputs The method of claim 1 or 2, comprising applying natural language processing techniques to the one or more test outputs generated by the target neural network for each test input.

4. identifying the one or more failing test inputs evaluating the one or more test outputs generated by the target neural network for each test input against one or more criteria.

4. The method according to any one of claims 1 to 3.

5. The method of claim 4 , wherein the one or more criteria specify that test output should not include offensive or biased content, protectable information processed during training, or personal contact information.

6. generating the segments based on the one or more failed test inputs, applying a text clustering algorithm to group various segments occurring in the one or more failed test inputs into a plurality of text clusters; generating the segment based on a segment included in one of the plurality of text clusters.

6. The method according to any one of claims 1 to 5.

7. using the plurality of segments to tune the target neural network that is deployed in the production environment.

7. The method according to any one of claims 1 to 6.

8. When executed by one or more computers, the one or more computers are caused to: maintaining a plurality of segments through a production environment in which a target neural network is deployed, the plurality of segments comprising: generating a plurality of test inputs by using a test case generation neural network; processing the plurality of test inputs using the target neural network to generate one or more outputs for each test input; identifying one or more failing test inputs that result in the target neural network producing a test output that does not meet one or more criteria based on the one or more test outputs produced by the target neural network for each test input; generating segments based on the one or more failed test inputs; and receiving online network input from a user through the production environment; determining, by the production environment, that the online network input includes at least one of the plurality of segments; in response, generating, by the production environment, default outputs in response to the online network inputs without processing the online network inputs using the target neural network to generate online network outputs; One or more computer-readable storage media storing instructions for performing operations including:

9. The computer-readable storage medium of claim 8 , wherein the production environment receives the plurality of segments from a pre-deployment environment system.

10. identifying the one or more failing test inputs applying natural language processing techniques to the one or more test outputs generated by the target neural network for each test input.

10. A computer-readable storage medium according to claim 8 or 9.

11. identifying the one or more failing test inputs evaluating the one or more test outputs generated by the target neural network for each test input against one or more criteria. A computer readable storage medium according to any one of claims 8 to 10.

12. 12. The computer-readable storage medium of claim 11, wherein the one or more criteria specify that test output should not include offensive or biased content, protectable information processed during training, or personal contact information.

13. generating the segments based on the one or more failed test inputs; applying a text clustering algorithm to group various segments occurring in the one or more failed test inputs into a plurality of text clusters; generating the segment based on a segment included in one of the plurality of text clusters.

13. A computer readable storage medium according to any one of claims 8 to 12.

14. 1. A system comprising one or more computers and one or more storage media storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to: maintaining a plurality of segments through a production environment in which a target neural network is deployed, the plurality of segments comprising: generating a plurality of test inputs by using a test case generation neural network; processing the plurality of test inputs using the target neural network to generate one or more outputs for each test input; identifying one or more failing test inputs that result in the target neural network producing a test output that does not meet one or more criteria based on the one or more test outputs produced by the target neural network for each test input; generating segments based on the one or more failed test inputs; and receiving online network input from a user through the production environment; determining, by the production environment, that the online network input includes at least one of the plurality of segments; in response, generating, by the production environment, default outputs in response to the online network inputs without processing the online network inputs using the target neural network to generate online network outputs; A system that causes a system to perform an operation including:

15. The system of claim 14 , wherein the production environment receives the plurality of segments from a pre-deployment environment system.

16. identifying the one or more failing test inputs applying natural language processing techniques to the one or more test outputs generated by the target neural network for each test input.

16. A system according to claim 14 or 15.

17. identifying the one or more failing test inputs evaluating the one or more test outputs generated by the target neural network for each test input against one or more criteria.

17. A system according to any one of claims 14 to 16.

18. 20. The system of claim 16, wherein the one or more criteria specify that test output should not include offensive or biased content, protectable information processed during training, or personal contact information.

19. generating the segments based on the one or more failed test inputs; applying a text clustering algorithm to group various segments occurring in the one or more failed test inputs into a plurality of text clusters; generating the segment based on a segment included in one of the plurality of text clusters.

19. A system according to any one of claims 14 to 18.

20. The operation further comprises: using the plurality of segments to tune the target neural network that is deployed in the production environment.

20. A system according to any one of claims 14 to 19.

Images