Improving performance of neural networks using automatically discovered failure cases

Abstract

The present disclosure relates to improving performance of a neural network using automatically discovered failing use cases. Methods, systems, and apparatus, including computer programs encoded on computer storage media, for adjusting a target neural network using automatically generated test cases prior to deploying the target neural network in a deployment environment. One of the methods can include generating a plurality of test inputs by using a test case generation neural network, processing the plurality of test inputs using the target neural network to generate one or more test outputs for each test input, and identifying, from the one or more test outputs generated by the target neural network for each test input, a failing test input that causes the target neural network to generate a test output that does not conform to one or more criteria.

Description

Improve neural network performance using automatically discovered failure cases.

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 303,958, filed January 27, 2022. The disclosure of that earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0003] This manual relates to the use of neural networks to process text. Background Technology

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output in response to a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates its output from the received input based on the current values ​​of its corresponding set of weights. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which can automatically generate test cases for a target neural network that is trained and configured to perform machine learning tasks on target network inputs. The target neural network can be any of various types and versions of neural networks generated for machine learning tasks. These neural networks may be based on different architectures, technologies, languages, vocabularies, etc., and may be trained using different training data.

[0006] After training, before deployment in a production environment, it may be necessary to validate and evaluate the target neural network for deployment suitability. For example, it might be necessary to evaluate the target neural network to determine whether, and if so, how often, the network output generated by the trained target neural network would fail to meet one or more criteria. Unlike traditional scenarios that rely on human annotators to manually write test cases for pre-deployment validation and evaluation, the system implementation uses a test case generation neural network to automatically generate test inputs.

[0007] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0008] Some of the techniques described can be used to automatically generate any number of test cases for a text generation neural network to identify potentially harmful or other undesirable behaviors of the network, such as outputting text containing offensive or confidential content. These techniques can then be used to analyze "failed" test cases to find common failure patterns. Appropriate measures can then be taken to improve the performance of the text generation neural network, for example, to prevent it from generating offensive or confidential content.

[0009] Harmful behaviors of neural networks have traditionally been identified by manually writing test cases using human annotators before deploying the network. Human annotation is expensive, for example, in terms of the cost associated with the human intelligence required to write test cases and the difficulty of scaling to the large number of test cases needed for complex target neural networks. Furthermore, the number and diversity of test cases are limited by human annotation. In contrast, the techniques described in this specification can generate a much larger number of test cases containing more diverse text, while requiring minimal human intervention, by automatically generating test cases using neural networks (which can sometimes be instantiated as another instance of a text generation neural network) and cueing engineering techniques.

[0010] Therefore, the described technology makes it easier to proactively identify and remediate potentially harmful network behaviors that were previously impossible or otherwise difficult to detect before deployment. The described technology facilitates several technological improvements. For example, the computational cost required to train and prepare text-generating neural networks for deployment can be reduced. As another example, the range of use cases for text-generating neural networks can be expanded. Thus, text-generating neural networks may become more suitable for deployment in production environments, such as within educational or healthcare organizations, where any harmful behavior of the deployed network could lead to serious consequences.

[0011] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 illustrates an example pre-deployment evaluation system and an example production environment.

[0013] Figure 2 is a diagram illustrating the performance evaluation of the target neural network relative to one or more standards.

[0014] Figure 3 is a flowchart of an example process for identifying failed test inputs.

[0015] Figure 4 is a flowchart of an example process for generating each test input using a test case generation neural network.

[0016] Figure 5 is a flowchart of an example process for identifying further failed test inputs.

[0017] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0018] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which can automatically generate test cases for a target neural network configured to perform machine learning tasks on target network inputs.

[0019] Figure 1 illustrates an example pre-deployment evaluation system 100 and an example production environment 140. The pre-deployment evaluation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and technologies described below are implemented.

[0020] The pre-deployment evaluation system 100 includes a target neural network 120. The target neural network 120 is an instance of a neural network trained to perform a machine learning task (i.e., process network inputs to generate network outputs for a machine learning task).

[0021] Typically, the target neural network 120 is configured to perform any of a variety of natural language modeling tasks. The following describes some examples of natural language modeling tasks that the target neural network 120 can be configured to perform.

[0022] As an example, the task could be a neural machine translation task. For instance, if the input to the neural network is a sequence of text—such as a sequence of words, phrases, characters, or word fragments in one language—the output generated by the neural network might be a translation of that text sequence into another language; that is, a sequence of text in another language that is translated from the input text sequence. As a specific example, the task could be a multilingual machine translation task, where a single neural network is configured to translate between multiple different source-target language pairs. In this example, the source language text can be augmented with an identifier that instructs the neural network to translate the source language text into the target language.

[0023] As another example, the task can be a natural language processing or understanding task, such as an implication task, paraphrasing task, text similarity task, sentiment task, sentence completion task, grammar task, etc., which operate on a text sequence of a natural language.

[0024] As another example, the task could be a text-to-speech task, where the input is natural language text or natural language text features, and the network output is a spectrogram or other data that defines the audio of the text spoken in natural language.

[0025] As another example, the task could be a text generation task, where the input is a sequence of texts and the output is another sequence of texts, such as the completion of the input text sequence, a response to a question posed in the input sequence, or a sequence of texts about a topic specified in the first text sequence.

[0026] As another example, the task could be a digital assistant task, where the input is a user request for assistance and the output includes information related to the user request. In implementation, this might involve generating (large) language models, particularly for conversations, such as a dialogue agent like Gopher. For example, the output could include instructions for a series of actions the user wants to perform, such as steps or subtasks of an overall task to achieve a technical effect and / or solve a technical problem in a real-world scenario. The instructions could be generated, for example, in the form of natural language (transmitted on-screen as sound and / or text).

[0027] In some cases, a natural language modeling task is a combination of multiple individual natural language modeling tasks; that is, the system is configured to perform multiple different individual natural language modeling tasks, such as two or more of the natural language modeling tasks described above. For example, the system may be configured to perform multiple individual natural language understanding tasks, where the network input includes identifiers of the individual natural language understanding tasks to be performed on the network input.

[0028] The target neural network 120 can have any of a variety of suitable network architectures that allow it to perform these natural language modeling tasks. For example, the target neural network 120 can be configured as a recurrent neural network (e.g., a neural network including one or more long short-term memory (LSTM) layers), an attention-based neural network (e.g., a neural network including one or more attention layers), or another generative neural network. As an illustration and not a limitation, some example architectures of the target neural network 120, as well as associated techniques for pre-training the target neural network 120 using unlabeled text training data, are described in more detail in Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher, arXiv:2112.11446, which are incorporated herein by reference in their entirety.

[0029] Training of the target neural network 120 can be performed locally on the pre-deployment evaluation system 100, or alternatively on another cloud-based training system. For example, the local or remote training system of the pre-deployment evaluation system 100 can use first unlabeled text training data to train the target neural network 120 on a first natural language modeling task to determine pre-trained values ​​for the parameters of the target neural network 120 (hereinafter referred to as "target network parameters"). Alternatively, the training system can further fine-tune the pre-trained target neural network 120 for one or more specific downstream tasks, for example, using labeled text training data suitable for a specific downstream task. The training system can train the target neural network 120 to converge an unsupervised loss function suitable for the first natural language modeling task, or until a fixed number of training steps have been performed.

[0030] Specifically, after training, the pre-deployment evaluation system 100 assesses the performance of the target neural network 120 for deployment in the production environment 140 by utilizing the test case generation neural network 110. Just as with the target neural network 120, the test case generation neural network 110 is an instance of a neural network that has already been trained to perform machine learning tasks.

[0031] For example, the training system can use second unlabeled text training data to train the test case generation neural network 110 on a second natural language modeling task to determine pre-trained values ​​for the parameters of the test case generation neural network 110 (hereinafter referred to as "test case network parameters"). The second natural language modeling task (and / or the second unlabeled text training data) may be the same as or different from the first natural language modeling task (and / or the first unlabeled text training data) on which the target neural network 120 has been trained.

[0032] The test case generating neural network 110 can have any of a variety of suitable network architectures similar to those of the target neural network 120. In fact, in some implementations, the test case generating neural network 110 and the target neural network 120 can have the same network architecture; for example, both can be configured as attention-based generative neural networks, including one or more attention layers. As used herein, attention layers are neural network layers that include attention mechanisms (e.g., multi-head self-attention mechanisms). Furthermore, in some of these implementations, networks 110 and 120 can be instantiated with a common set of network parameter values; for example, both can be instantiated with the same pre-trained parameter values ​​determined by training on the same natural language modeling task using the same unlabeled text training dataset.

[0033] At a high level, the pre-deployment evaluation system 100 uses a test case generation neural network 110 to automatically generate multiple test inputs 112, which are then processed by a target neural network 120 according to target network parameters to generate test outputs 122. The pre-deployment evaluation system 100 then evaluates these test outputs 122 against one or more criteria 131 to determine whether the target neural network 120 is suitable for deployment in the production environment 140.

[0034] In some implementations, the pre-deployment evaluation system 100 may output information to a user regarding the evaluation results, indicating whether the target neural network 120 is suitable for deployment. In some implementations, if the target neural network 120 fails to meet one or more of the criteria during the pre-deployment evaluation, the pre-deployment evaluation system 100 may adjust the network accordingly to improve its suitability for one or more criteria; if the target neural network 120 (or, a adjusted target neural network obtained by adjusting the target neural network 120 using the techniques broadly described in this specification) passes all criteria, the pre-deployment evaluation system 100 may provide data for the specified target neural network 120 (or the adjusted target neural network) to the production environment 140 to allow the production environment 140 to deploy the target neural network 120 (or the adjusted target neural network) for performing inference for machine learning tasks, i.e., generating online network outputs for machine learning tasks for online network inputs.

[0035] As used in this article, a production environment refers to an environment where neural networks can be applied to real-time user input data rather than training or test data with known expected results. Input data in a production environment can be referred to as online data, in contrast to offline training or test data.

[0036] Production environment 140 can be implemented on one or more computing devices. For example, production environment 140 can deploy the target neural network 120 (or a modified target neural network) in a data center including hundreds of computers to process input received from remote users, or the training parameter values ​​of the target neural network 120 can be provided to edge devices, such as mobile phones, smart personal assistant devices, smartwatches, smart displays, or other IoT devices, via wired or wireless network connections so that the target neural network 120 can be used to perform machine learning tasks on edge devices.

[0037] The pre-deployment evaluation system 100 implements appropriate prompting techniques to effectively guide the generation process of test inputs 112 using a test case generation neural network 110. Incorporating prompting techniques into the test case generation process can be advantageous for several reasons. On one hand, prompts improve the controllability of the evaluation process, enabling faster identification of various failing test cases. On the other hand, prompts guide the test case generation neural network 110 to generate specific types of test inputs, including test inputs with text rarely appearing in the training data, thereby aiding in the identification and analysis of specific failure modes of the target neural network 120.

[0038] The test case generation neural network 110 is therefore configured to receive test case network input 102 as or including natural language prompts, and to process the test case network input 102 according to the test case network parameters to generate a test case network output from which test cases can be generated. Test cases include text input 112, which can then be provided for processing by the target neural network 120. Natural language prompts are used to guide the test case generation neural network 110 to generate test cases with different levels of diversity and complexity. The generation of natural language prompts as test case network input and the generation of text input from the test case network output will be discussed in more detail below with reference to Figures 3-4.

[0039] Each test input 112 may include text. As used herein, text can refer to natural language text, which includes one or more text elements (e.g., one or more words in a sentence) of one or more natural languages ​​(including English, Chinese, Japanese, and Korean, to name just a few). Text can also refer to computer-readable text, which includes one or more computer code elements of one or more computer programming languages ​​(e.g., Python, C++, C#, Java, Ruby, PHP, etc.).

[0040] Test inputs generated in this way are relatively inexpensive (e.g., in terms of the cost associated with the human intelligence required to write test cases, the cost associated with the additional computational resources required to implement and train individual test case generation models, or both), and they cover a wide range of different topics with varying levels of testing difficulty, making it easier to discover any potentially harmful behaviors of the target neural network being trained, behaviors that may differ from or be impossible to discover manually. Furthermore, unlike other traditional systems that attempt to evaluate the target neural network by brute force, such as by submitting gibberish or other incomprehensible content, test inputs generated in this way include text that appears plausible in natural language and thus mimics those that users might submit once deployed in production.

[0041] The pre-deployment evaluation system 100 provides test inputs 112 to the target neural network 120 and uses the target neural network 120 to process each test input 112 according to the target network parameters to generate one or more test outputs 122 for the test input 112. Each test output 122 may similarly include text. The test outputs 122 can be evaluated relative to any of the various standards 131 maintained by the system 100 that correspond to the production environment 140. The data defining the standards 131 can be uploaded by users of the system 100, or alternatively, can be determined by the system 100 itself based on the known requirements of the production environment 140. The standards 131 can be stored in a data store associated with different production environments. The standards 131 can be revised and updated periodically as needed.

[0042] For example, one or more standards 131 may specify that the text included in the test output should not include certain content or information. For instance, a standard may specify that the test output should not include one or more of the following: offensive content, error messages, or confidential or private information, such as personal contact information. As another example, a standard may specify that the test output should not lead to undesirable data leakage. For example, in the context of federated learning (where different organizations train a network together using each organization's private local data), text from any private training data or other protected data (e.g., copyrighted data) used to pre-train the target neural network 120 should not be included in the test output. In this example, the data security of the target neural network can be increased because text from private training data or other protected data is not included in the online network output to be transmitted to the user.

[0043] To automatically determine whether these test outputs 122 do not meet one or more criteria, the pre-deployment evaluation system 100 may also include a text-based classifier engine 130 that processes each test output 122 to determine whether the text output 122 does not meet one or more criteria. In some implementations, the text-based classifier engine 130 may implement: a text classifier machine learning model, such as a neural network, logistic regression model, support vector machine (SVM), or decision tree or random forest model; or another black-box text classifier configured to process the test outputs 122 to generate a predicted probability that the text output 122 will not meet one or more criteria as classifier output 132.

[0044] As an illustration and not a limitation, some example architectures of attention-based classification neural networks and associated techniques for training such networks are described in more detail in Xu et al., Botadversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968, which are hereby incorporated in their entirety by reference.

[0045] In some implementations, the text-based classifier engine 130 may implement a deterministic text-based classification algorithm that generates a binary classifier output 132 indicating whether the text output 122 will not meet one or more criteria. Such a deterministic algorithm can typically be defined using high-level computer programming code (e.g., Python or C++ code).

[0046] For example, if a single criterion exists, classifier engine 130 can generate a probability score indicating that test output 122 does not meet the criterion based on the output and optional additional information (e.g., test input 112 or other context from the dialogue history with target neural network 120). As another example, if multiple criteria exist, classifier engine 130 can include separate classifiers for different criteria, and if test output 122 does not meet any of the criteria, classifier engine 130 can generate a failure indication. As yet another example, classifier engine 130 can include a single classifier that makes classification predictions for all criteria 131.

[0047] Figure 2 is an illustration of evaluating the performance of a target neural network relative to one or more criteria. In the example of Figure 2, test outputs generated by the target neural network that contain one or more of the following: offensive or biased content, protected information processed during training, or personal contact information, will fail one or more of the criteria.

[0048] Some implementations of the pre-deployment evaluation system 100 can use identified failed test inputs, i.e., test inputs 112 from which the target neural network 120 generates test outputs 122 that do not meet one or more criteria, to adjust the target neural network 120 so that adjusted instances of the target neural network 120 can generate test outputs that are unlikely to fail to meet one or more criteria. This improves the applicability of the adjusted target neural network 120 for deployment in the production environment 140.

[0049] In some of these implementations, the pre-deployment evaluation system 100 may tune the target neural network 120 by applying natural language processing techniques to analyze the identified failed test inputs to determine specific text content that, when included in the test inputs, might cause the target neural network 120 to generate test outputs that do not meet one or more criteria. For example, a text clustering algorithm may be applied to group different text segments appearing in the failed test inputs into multiple text clusters. A text segment may be a word or a phrase comprising multiple words or other terms from a vocabulary. From these multiple text clusters, one or more of the most frequently occurring specific text segments—or in other words, those most likely to generate test outputs that do not meet one or more criteria—can then be identified.

[0050] The pre-deployment evaluation system 100 can then adjust the target neural network 120 based on taking appropriate measures to account for such content. For example, training examples from unlabeled text training data containing specific text content can be removed so that they will not be processed by the target neural network 120 during training. They will also not be used to calculate the loss for updating the target network parameter values. As another example, training examples containing specific text content can be adjusted or otherwise modified before being processed by the target neural network 120, for example, by removing specific text segments from the training examples or by adding additional text segments to the training examples.

[0051] Additionally or alternatively, some implementations of the pre-deployment evaluation system 100 may use identified failed test inputs to improve the suitability of the target neural network 120 after deployment in the production environment 140. For example, the system 100 may do this by sending instructions to one or more computing devices implementing the production environment 140 to configure the target neural network 120 to refuse to provide any online network output or generate default output in response to user-submitted online network input containing such content.

[0052] Figure 3 is a flowchart of an example process 300 for identifying failed test inputs. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system properly programmed according to this specification, such as the pre-deployment evaluation system 100 of Figure 1, can execute process 300.

[0053] The system generates multiple test inputs using a test case generation neural network with multiple test case network parameters (step 302). Each test input can contain text. In some implementations, the test case generation neural network can be a pre-trained generative neural network model.

[0054] For example, the test case generation neural network can be an autoregressive generative model that has been pre-trained on a first natural language modeling task using first unlabeled text training data. In this example, the test case network output, which generates one or more test inputs based on it, can include the corresponding test case network score for text tokens (or simply "tokens") in a candidate token vocabulary at each of multiple test case network output time steps, where each token can represent one or more of the following: characters, word fragments, words, punctuation marks, etc. The test case generation neural network can be configured to autoregressively generate test case network scores, for example, by processing the test case network inputs according to multiple test case network parameters in such a way as follows: conditioned on the score distributions of one or more previous test case network output time steps, it generates a score distribution on the candidate token vocabulary at each of the multiple test case network output time steps, which includes the corresponding test case network score for each candidate token. In particular, the test case network input can include natural language prompts. The generation of such natural language prompts will be discussed below with reference to Figure 4.

[0055] The system processes multiple test inputs using a target neural network with multiple target network parameters to generate one or more test outputs for each test input (step 304). Each test output may similarly contain text. In some implementations, the target neural network may be a pre-trained generative neural network model. Furthermore, in some implementations, the test case generating neural network and the target neural network may have the same network architecture and may be trained on the same task using the same training data.

[0056] For example, the target neural network could be an autoregressive generative model that has been pre-trained on a second natural language modeling task using second unlabeled text training data. This second natural language modeling task (and / or the second unlabeled text training data) may be the same as or different from the first natural language modeling task (and / or the first unlabeled text training data). In this example, the target neural network could be configured to process each test input according to multiple target network parameters to generate a target network output that specifies a corresponding test output for each test input.

[0057] Similar to test case generation neural networks, target neural networks can achieve this by generating a corresponding target network score for each candidate lexicon in the candidate lexicon vocabulary at each of the multiple target network output time steps, and then using these scores to select the lexicon for that time step. The system can then generate test outputs based on the connections to these selected lexicons.

[0058] The system identifies failed test inputs that cause the target neural network to generate test outputs that do not conform to one or more criteria (step 306). Specifically, the system may, for example, evaluate the test outputs relative to any of a variety of criteria using one or more text-based classifiers to determine which or which particular test outputs do not conform to the criteria, and accordingly identify the test inputs from which the target neural network generates specific test outputs.

[0059] Figure 4 is a flowchart of an example process 400 for generating each test input using a test case generation neural network. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system properly programmed according to this specification, such as the pre-deployment evaluation system 100 of Figure 1, can execute process 400.

[0060] The system generates natural language prompts as network input for test cases (step 402). Natural language prompts may include text, numbers, punctuation marks, or combinations thereof, and possibly other information. For example, a natural language prompt may be a sentence describing a given topic using multiple words, generated based on a test case template. The test case template may define fixed text segments that will be included in multiple natural language prompts, as well as variable text segments that differ across the multiple natural language prompts.

[0061] In some implementations, the system can generate natural language prompts by iteratively accessing a list of given topics known to be associated with the production environment. In some implementations, the system can generate natural language prompts based on previously generated natural language prompts, for example, by reusing specific text segments (or content similar to specific text segments) from previously generated specific natural language prompts.

[0062] The system uses a test case generation neural network to process the test case network input based on multiple test case network parameters, and then autoregressively generates the test case network output at multiple test case network output time steps (step 404). At each of the multiple test case network output time steps, the test case network output includes the corresponding test case network score for each candidate lexicon in the candidate lexicon vocabulary.

[0063] The system generates test input based on sampled lexical units according to the test case network scores contained in the test case network output (step 406). Specifically, the system can repeatedly sample lexical units from the candidate lexical vocabulary based on the test case network scores corresponding to each of the multiple test case network output time steps. Through sampling, the system can generate more than one text input from a single test case network output, i.e., by sampling different lexical units to include in different test inputs.

[0064] Some implementations of this system can employ more sophisticated lexical sampling techniques to improve the quality of test inputs. As an example, at each of the multiple test case network output time steps, the system can sample lexical units from a subset of candidate lexical units in the vocabulary (rather than the entire vocabulary) as lexical units to be included in the test input. For example, this subset could include candidate lexical units for which the test case neural network has generated test case network scores greater than a certain threshold, such as test case network scores in the top 90% or top 95% of all scores. As another example, the system can repeatedly sample different lexical units to include in different test inputs until a predetermined number of different test inputs have been generated, each meeting a validity criterion. For example, the validity criterion could specify that each test input should include a required terminating string or otherwise meet the terminating criterion.

[0065] Typically, process 300 can be repeated multiple times as needed to evaluate the performance of the target neural network that has undergone multiple training steps. For example, process 300 can be repeated hourly, daily, or weekly during the training process of the target neural network. Process 300 can also be triggered before the target neural network is deployed in a production environment.

[0066] After one or more iterations of process 300, some implementations of the system may adjust the values ​​of the test case network parameters of the test case generation neural network based on knowledge gained from previous iterations of process 300, in order to update the way test cases for the target neural network are generated by using the test case generation neural network, thereby improving the diversity, complexity, or both of the test cases generated in the next iteration of process 300.

[0067] As an example, supervised learning (SL) techniques can be used. Supervised learning techniques evaluate a learning objective that depends on the total number of failed test inputs generated in one or more previous iterations of process 300 using a test case generation neural network, such as log-likelihood. The system can use this supervised learning technique to fine-tune pre-trained values ​​of multiple test case network parameters, modifying the test case generation neural network in the way it generates test case network outputs, making it more likely to generate failed test inputs from those outputs in the next iteration of process 300.

[0068] As another example, reinforcement learning (RL) techniques can be used. The system can use such reinforcement learning to (i) further fine-tune the tuning values ​​or (ii) fine-tune the pre-trained values ​​of multiple test case network parameters, such that the test case generating neural network is modified in the way it generates the test case network output, making it more likely to generate failing test inputs from that output in the next iteration of process 300. For example, the reinforcement learning technique could be an actor-critic technique that optimizes the reinforcement learning loss based on the diversity of the test case network output, where the optimized reinforcement learning loss includes a loss term such as a Kullback–Leibler (KL) divergence term.

[0069] As another example, the subsampling technique described in Figure 5 can be used.

[0070] Figure 5 is a flowchart of an example process 500 for identifying additional failed test inputs using subsampling techniques. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system properly programmed according to this specification, such as the pre-deployment evaluation system 100 of Figure 1, can execute process 500.

[0071] For example, the system can execute process 500 after one or more iterations of process 300, using one or both of SL or RL techniques to adjust the test cases to generate a neural network.

[0072] The system generates multiple additional test inputs (step 502). The system can repeat sub-steps 504-510 of step 502 as described below to generate each additional test input.

[0073] The system samples one or more test inputs from a plurality of test inputs already generated using a test case generation neural network (step 504). In some implementations, the system may sample one or more test inputs already generated from one or more previous iterations of process 300 with consistent randomness. In other implementations, the system may sample one or more test inputs from a distribution that specifies a greater probability of sampling a failed test input than sampling other test inputs from a plurality of test inputs that are not failed test inputs.

[0074] The system generates another test case network input that includes one or more sampled test inputs (step 506). In some implementations, the system may use text contained in one or more sampled test inputs as another test case network input, while in other implementations, the system may append or prepend text contained in one or more sampled test inputs to a natural language prompt to form another test case network input.

[0075] The system uses a test case generation neural network to process the input of another test case network based on the network parameters of multiple test cases to generate another test case network output (step 508).

[0076] The system generates additional test input based on sampled terms, according to the test case network score contained in the output of another test case network (step 510). Steps 508 and 510 are similar to steps 404 and 406 in Figure 4 (although they are performed based on test case network input generated in a different manner).

[0077] The system uses the target neural network to process multiple additional test inputs to identify further failed test inputs (step 512). Step 512 is similar to step 306 in Figure 3.

[0078] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0079] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0080] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or multiple processors or computers. The device may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0081] Computer programs can be written in any programming language (also referred to as or described as programs, software, software applications, apps, modules, software modules, scripts, or code), including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not need to, correspond to a file in a file system. A program may be stored as a part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0082] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or not structured at all, and which can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0083] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0084] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (such as an FPGA or ASIC) or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0085] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices, or operatively coupled to receive data from or transfer data to one or more mass storage devices, or both, for storing data, such as disks, magneto-optical disks, or optical disks. However, a computer does not require such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0086] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0087] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and indicating device, such as a mouse or trackball, through which the user provides input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in return.

[0088] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0089] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX.

[0090] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components such as data servers, middleware components such as application servers, front-end components such as client computers with graphical user interfaces, web browsers, or apps, or any combination of one or more such back-end, middleware, or front-end components through which a user can interact with embodiments of the subject matter described in this specification via the graphical user interface, web browser, or app. The components of the system can be interconnected via digital data communication of any form or medium—e.g., a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”) (e.g., the Internet).

[0091] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on separate computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the user device.

[0092] While this specification contains numerous specific details of implementation, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0093] Similarly, although operations are depicted in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0094] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, comprising: Multiple test inputs are generated by using a test case generation neural network with multiple test case network parameters; The target neural network is used to process the plurality of test inputs with a plurality of target network parameters to generate one or more test outputs for each test input, wherein the target neural network is configured to process each test input according to the plurality of target network parameters to generate a target network output, the target network output specifying a corresponding test output for the test input; and to identify, from the one or more test outputs generated by the target neural network for each test input, failed test inputs that cause the target neural network to generate test outputs that do not conform to one or more criteria.

2. The method according to claim 1, wherein: Each of the test inputs and the test outputs includes text; the test case generating neural network is a generative neural network that is pre-trained on a first natural language modeling task using first unlabeled text training data and configured to generate test case network outputs by generating a corresponding test case network score for each candidate noun in the candidate noun vocabulary at each of the multiple test case network output time steps; The target neural network is another generative neural network that is pre-trained on a second natural language modeling task using second unlabeled text training data and configured to generate the target network output by generating the corresponding target network score for each candidate lexicon in the candidate lexicon vocabulary at each of the multiple target network output time steps.

3. The method according to claim 2, wherein: The test case generating neural network and the target neural network have the same network architecture, the first natural language modeling task and the second natural language modeling task are the same natural language modeling task, and the first unlabeled text training data and the second unlabeled text training data are the same unlabeled text training data.

4. The method according to claim 2, wherein, Generating each test input using the test case generation neural network includes: generating natural language prompts as test case network inputs, the natural language prompts including text, numbers, punctuation marks, or combinations thereof; processing the test case network inputs using the test case generation neural network according to the plurality of test case network parameters to generate the test case network outputs, the test case network outputs containing the corresponding test case network score for each candidate lexicographical term in the candidate lexicographical vocabulary at each of the plurality of test case network output time steps; and generating the test inputs based on sampled lexics according to the respective test case network scores contained in the test case network outputs.

5. The method according to claim 4, wherein, Generating the test input based on the test case network score contained in the test case network output, based on sampled lexical units, includes, at each of some of the multiple test case network output time steps: sampling a sampled lexical unit from a subset of candidate lexical units as a lexical unit to be included in the test input, for which the test case generation neural network has generated a test case network score greater than a certain threshold.

6. The method according to claim 4, wherein, Generating the test input based on the test case network score contained in the test case network output, based on sampled lexical units, includes: repeatedly sampling different lexical units to include in different test inputs until a predetermined number of different test inputs have been generated, each of which satisfies a validity criterion.

7. The method according to claim 1, further comprising: Each of the multiple additional test inputs is generated by: sampling one or more test inputs from the multiple test inputs already generated using the test case generation neural network; generating another test case network input including the sampled one or more test inputs; processing the other test case network input using the test case generation neural network according to the multiple test case network parameters to generate another test case network output; generating the additional test input based on sampled lexical units according to the test case network scores contained in the other test case network output; and processing the multiple additional test inputs using the target neural network to identify further failed test inputs.

8. The method according to claim 7, wherein, Sampling the one or more test inputs from the plurality of test inputs includes sampling the one or more test inputs from a distribution that specifies a greater probability of sampling the failed test inputs than sampling other test inputs from the plurality of test inputs that are not failed test inputs.

9. The method of claim 7, further comprising, before generating each of the plurality of additional test inputs: using supervised learning techniques to fine-tune pre-trained values ​​of the parameters of the plurality of test case networks to cause the test case generating neural network to generate test case network outputs from which the failed test inputs are more likely to be generated.

10. The method of claim 7, further comprising, before generating each of the plurality of additional test inputs: Reinforcement learning techniques are used to fine-tune the pre-trained values ​​of the multiple test case network parameters, so that the test case generation neural network generates test case network outputs from which the failed test inputs are more likely to be generated.

11. The method according to claim 10, wherein, The reinforcement learning technique includes an actor-critic technique that optimizes the reinforcement learning loss, which includes a loss term that depends on the diversity of the network output of the test cases.

12. The method according to claim 1, wherein, The one or more criteria specify that the test output should not contain one or more of the following: offensive content, error messages, confidential or private information, text from any private training data used to pretrain the target neural network, and text from any protected training data used to pretrain the target neural network.

13. The method according to claim 1, wherein, Identifying the failed test inputs includes: processing each test output generated by the target neural network using a text classifier neural network to generate a prediction probability that the test output will not meet one or more criteria.

14. The method according to claim 1, wherein, Identifying the failed test inputs includes processing each test output generated by the target neural network using a text-based classifier to determine whether the test output does not conform to one or more criteria.

15. The method according to claim 14, wherein, The text-based classifier includes a black-box text classifier or a deterministic text-based classification algorithm.

16. The method of any one of claims 1-15, further comprising using the identified failed test inputs to adjust the target neural network to induce the target neural network to generate test outputs that are unlikely to fail to meet the one or more criteria.

17. The method according to claim 16, wherein, Adjusting the target neural network using the identified failed test inputs includes: using natural language processing techniques incorporating text clustering to analyze the identified failed test inputs to determine text segments that, when included in the test inputs, have the highest probability of causing the target neural network to generate test outputs that do not conform to the one or more criteria.

18. The method according to claim 16, wherein, Adjusting the target neural network using the identified failed test input includes one or more of the following: removing specific training examples from unlabeled text training data; and adjusting the target network input before processing the target network input of the target neural network using the target neural network, the adjustment including removing a first text segment from the target network input or adding a second text segment to the target network input.

19. A computer-readable storage medium for storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform corresponding operations in the method of any one of claims 1-18.

20. A system comprising one or more computers and one or more storage devices storing instructions, wherein, when executed by the one or more computers, the instructions cause the one or more computers to perform a corresponding operation in the method of any one of claims 1-18.

Images