System and method for debugging neural networks using coverage-guided fuzz testing

Abstract

The present disclosure provides systems and methods for debugging neural networks. In one example, a computer-implemented method is provided, comprising obtaining, by one or more computing devices, one or more inputs from an input corpus. The method further comprises mutating, by the one or more computing devices, the one or more inputs and providing the one or more mutated inputs to the neural network; obtaining, by the one or more computing devices, a set of covering arrays as a result of the neural network processing the one or more mutated inputs; determining, by the one or more computing devices, at least in part based on the set of covering arrays, whether the one or more mutated inputs provide new coverage; and upon determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus.

Description

[0001] This application is based upon and claims the benefit of U.S. Provisional Application No. 62 / 673,751, having a filing date of May 18, 2018, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0002] The present disclosure relates generally to machine learning models and more specifically to discovering unexpected behavior in neural networks. Background Art

[0003] The use of machine learning models, such as neural networks, is becoming increasingly important in solving a variety of tasks that have traditionally been difficult for computing systems. However, machine learning models are generally difficult to interpret and debug. As machine learning models, such as neural networks, become more common, it becomes more desirable to test neural networks to discover bugs and / or other undesirable behaviors before implementing them in the "real world." Summary of the Invention

[0004] Aspects and advantages of the embodiments of the present disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0005] One example aspect of the present disclosure is directed to debugging a neural network. The method may include obtaining, by one or more computing devices, one or more inputs from an input corpus. The method may also include mutating, by the one or more computing devices, the one or more inputs. The method may also include providing, by the one or more computing devices, the one or more mutated inputs to the neural network. The method may also include obtaining, by the one or more computing devices, a set of coverage arrays as a result of the neural network processing the one or more mutated inputs, the set of coverage arrays describing whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network. The method may also include determining, by the one or more computing devices, based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage. The method may also include, after determining that the one or more mutated inputs provide new coverage, adding, by the one or more computing devices, the one or more mutated inputs to the input corpus.

[0006] In some embodiments, the method may further include obtaining, by one or more computing devices, a set of metadata arrays as a result of the neural network processing the one or more mutated inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determining, by the one or more computing devices, whether an objective function is satisfied based at least in part on the set of metadata arrays; and upon determining that the objective function is satisfied, adding, by the one or more computing devices, the one or more mutated inputs to a list of test cases.

[0007] In some embodiments, the method may also include determining, by one or more computing devices, whether one or more mutated inputs provide new coverage based at least in part on the set of covering arrays, including: generating, by one or more computing devices, an activation vector based at least in part on the set of covering arrays; executing, by one or more computing devices, an approximate nearest neighbor algorithm to identify a previous activation vector; determining, by one or more computing devices, a distance between the activation vector and the previous activation vector identified by the approximate nearest neighbor algorithm; and comparing, by one or more computing devices, the distance to a threshold distance; wherein, when the distance is greater than the threshold distance, the one or more mutated inputs provide new coverage.

[0008] In some implementations, obtaining one or more inputs from the input corpus can include selecting one or more inputs from the input corpus using uniform random selection.

[0009] In some implementations, obtaining one or more inputs from an input corpus may include using The heuristic selects one or more inputs from the input corpus, where p(c k ,t) gives the selection of input corpus element c at time t k The probability of t k is element c k The time at which the sentence was added to the input corpus.

[0010] In some implementations, mutating one or more inputs can include adding white noise having a user-configurable variance to the one or more inputs.

[0011] In some embodiments, mutating one or more inputs may include adding white noise having a user-configurable variance to the one or more inputs, wherein the difference between the mutated inputs and the original inputs from which the mutated inputs descended is constrained to have a user-configurable L ∞ norm.

[0012] In some implementations, determining whether one or more mutated inputs provide new coverage can include determining whether the neural network has reached a new state that it had not previously reached.

[0013] In some implementations, determining whether the neural network has reached a new state that it has not previously reached can include determining whether an activation vector is approximately close to a previous activation vector.

[0014] In some implementations, determining whether the objective function is satisfied can include determining whether the neural network has reached a desired state.

[0015] In some implementations, the desired state may be an error state of the neural network.

[0016] Another example aspect of the present disclosure is directed to a computing device. The computing device may include one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing device to perform operations. The instructions, when executed, may cause the computing device to obtain one or more inputs from an input corpus. The instructions, when executed, may also cause the computing device to mutate the one or more inputs. The instructions, when executed, may also cause the computing device to provide the one or more mutated inputs to a neural network. The instructions, when executed, may also cause the computing device to obtain a set of covering arrays as a result of the neural network processing the one or more mutated inputs, the set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network. The instructions, when executed, may also cause the computing device to determine, based at least in part on the set of covering arrays, whether the one or more mutated inputs provide new coverage. The instructions, when executed, may also cause the computing device to add the one or more mutated inputs to the input corpus upon determining that the one or more mutated inputs provide new coverage.

[0017] In some embodiments, the computing device may further include instructions that, when executed, cause the computing device to obtain a set of metadata arrays as a result of the neural network processing the one or more mutated inputs, the set of metadata arrays describing metadata associated with the execution of the neural network processing the one or more mutated inputs; determine whether an objective function is satisfied based at least in part on the set of metadata arrays; and upon determining that the objective function is satisfied, add the one or more mutated inputs to a list of test cases.

[0018] In some implementations, the computing device may further include instructions that, when executed, cause the computing device to obtain an input corpus from a seed corpus, the seed corpus containing at least one set of inputs.

[0019] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations. The operations include obtaining one or more inputs from an input corpus. The operations also include mutating the one or more inputs. The operations also include providing the one or more mutated inputs to a neural network. The operations also include obtaining a set of covering arrays as a result of the neural network processing the one or more mutated inputs, the set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network. The operations also include determining, based at least in part on the set of covering arrays, whether the one or more mutated inputs provide new coverage. The operations also include adding the one or more mutated inputs to the input corpus after determining that the one or more mutated inputs provide new coverage.

[0020] In some embodiments, one or more non-transitory computer-readable media may store instructions that, when executed by one or more processors of a computing system, cause the computing system to obtain a set of metadata arrays as a result of a neural network processing one or more mutated inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determine whether an objective function is satisfied based at least in part on the set of metadata arrays; and upon determining that the objective function is satisfied, add the one or more mutated inputs to a list of test cases.

[0021] In some embodiments, a computer-implemented method for debugging a neural network may include obtaining, by one or more computing devices, one or more inputs from an input corpus; mutating, by the one or more computing devices, the one or more inputs; providing, by the one or more computing devices, the one or more mutated inputs to the neural network; obtaining, by the one or more computing devices, a set of metadata arrays as a result of the neural network processing the one or more mutated inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determining, by the one or more computing devices based at least in part on the set of metadata arrays, whether a target function is satisfied; and upon determining that the target function is satisfied, adding, by the one or more computing devices, the one or more mutated inputs to a list of test cases.

[0022] In some embodiments, the computer-implemented method may further include providing, by one or more computing devices, one or more variant inputs of the input corpus to the neural network; and obtaining, by the one or more computing devices, a set of covering arrays as a result of the neural network processing the one or more variant inputs, the set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more variant inputs by the neural network.

[0023] In some embodiments, the computer-implemented method may further include determining, by the one or more computing devices, based at least in part on the set of covering arrays, whether the one or more mutated inputs provide new coverage; and upon determining that the one or more mutated inputs provide new coverage, adding, by the one or more computing devices, the one or more mutated inputs to the input corpus.

[0024] In some implementations, a computer-implemented method for debugging a neural network may include providing, by one or more computing devices, one or more variant inputs of an input corpus to the neural network; and obtaining, by the one or more computing devices, a set of metadata arrays as a result of the neural network processing the one or more variant inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more variant inputs.

[0025] In some embodiments, the computer-implemented method may further include determining, by one or more computing devices, whether an objective function is satisfied based at least in part on the set of metadata arrays; and after determining that the objective function is satisfied, adding, by the one or more computing devices, one or more mutated inputs to a list of test cases.

[0026] Neural networks can be configured to receive any kind of numerical data input and generate any kind of score, classification, or regression output based on the input.

[0027] For example, if the input to a neural network is an image or features that have been extracted from an image, then the output generated by the neural network for a given image can be a score for each class in a set of object classes, each score representing an estimated likelihood that the image contains an object belonging to that class.

[0028] As another example, if the input to the neural network is internet resources (e.g., web pages), documents or portions of documents, or features extracted from internet resources, documents or portions of documents, then the output generated by the neural network for a given internet resource, document or portion of a document can be a score for each topic in a set of topics, each score representing an estimated likelihood that the internet resource, document or portion of a document is about that topic.

[0029] As another example, if the input to a neural network is features of the impression context of a particular ad, then the output generated by the neural network may be a score representing the estimated likelihood that the particular ad will be clicked.

[0030] As another example, if the input to the neural network is features of personalized recommendations for a user, e.g., features characterizing the context of the recommendations, e.g., features characterizing actions previously taken by the user, then the output generated by the neural network can be a score for each content item in a set of content items, each score representing an estimated likelihood that the user will respond favorably to the recommended content item.

[0031] As another example, if the input to a neural network is a sequence of text in one language, the output generated by the neural network can be a score for each text segment in a set of text segments in another language, each score representing an estimated likelihood that the text segment in the other language is a correct translation of the input text into the other language.

[0032] As another example, if the input to a neural network is a sequence representing a spoken utterance, the output generated by the neural network may be a score for each text segment in a set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of the utterance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification, which refers to the accompanying drawings, in which:

[0034] Figure 1 Depicted is a block diagram of an example computing system that may be used with a machine learning model according to an example embodiment of the present disclosure.

[0035] Figure 2 Depicted is a block diagram of an example coverage guided fuzzing system according to an example embodiment of the present disclosure.

[0036] Figure 3 Depicted is a flowchart of example operations for performing neural network debugging, according to an example embodiment of the present disclosure.

[0037] Repeated reference numerals across the various drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0038] Overview

[0039] In general, the present disclosure is directed to machine learning models, such as neural networks. In particular, the systems and methods of the present disclosure can provide, for example, testing neural networks to discover defects (bugs) and / or other unexpected behaviors before they are deployed. According to one aspect of the present disclosure, coverage-guided fuzz testing can be applied to neural networks to allow debugging of neural networks. For example, in some embodiments, coverage-guided fuzz testing can be applied to neural networks to provide for discovering numerical errors in trained neural networks, discrepancies between generated neural networks and quantized versions of those networks, unexpected behaviors in visualization models (e.g., character-level language models, etc.), etc.

[0040] Machine learning models can be difficult to debug or explain for a variety of reasons, ranging from the conceptual difficulty of specifying in formal terms what a user wants to know about the model, to the statistical and computational difficulties of obtaining answers to formally specified questions. Neural networks can be particularly difficult to debug because even relatively simple formal questions about them are computationally expensive to answer, and because software implementations of neural networks can deviate significantly from the theoretical model.

[0041] In general, coverage-guided fuzz testing provides for maintaining an input corpus that includes the inputs to the program under consideration. Those inputs are randomly mutated according to some mutation procedure, and when a mutated input exercises new coverage (e.g., causes the code to execute in a different way than previously seen, etc.), the mutated input is added to the input corpus.

[0042] The systems and methods of the present disclosure provide for random mutation of inputs to a neural network, wherein the mutations are guided by a coverage metric toward the goal of satisfying user-specified constraints. According to one aspect of the present disclosure, coverage can be measured by analyzing activation vectors of a neural network coverage graph. For example, in some embodiments, new coverage can be determined based on whether the neural network has led to a state that the neural network had not previously reached, such that the new coverage helps provide incremental progress in debugging.

[0043] As an example, fast approximate nearest neighbor algorithms can be used to determine whether two sets of neural network "activations" are meaningfully different from each other. This provides a coverage metric that produces useful results for the neural network even when the underlying implementation of the neural network does not use many data-dependent branches. For example, in some embodiments, the activations associated with each input (or some subset of them) can be stored and examined to determine whether coverage has increased on a given input by using an approximate nearest neighbor algorithm to see if there are any other sets of activations within a pre-specified distance.

[0044] According to one aspect of the present disclosure, in some embodiments, coverage-guided fuzz testing of a neural network can begin with a seed corpus containing at least one set of inputs for a computation graph. The inputs can be restricted to those that are, in some sense, valid neural network inputs. For example, if the input is an image, the inputs can be restricted to those that have the correct size and shape and are located in the same interval as the input pixels of the dataset under consideration. As another example, if the input is a sequence of characters, the inputs can be restricted to characters in a vocabulary extracted from a training set.

[0045] Given this seed corpus, the neural network debugging system can select elements from the input corpus according to some heuristic (e.g., uniform random selection, some defined probabilistic heuristic, etc.) until it is instructed to stop or some other stopping criterion is met. Given this input, the neural network debugging system can perform some modification on that input. For example, the modification can be as simple as simply flipping the sign of the input pixel in the image. Additionally or alternatively, it can also be constrained to follow some constraint on the total modification of the corpus elements over time. The mutated input can then be fed to the neural network. In some embodiments, two things can be extracted from the neural network: a set of covering arrays from which the actual covering can be computed, and a set of metadata arrays from which the results of the objective function can be computed. For example, the covering arrays can describe which neurons of the neural network were activated during the processing of the input and can therefore be referred to as or used to generate "activation vectors." As another example, the metadata arrays can describe the behavior, output, results, predictions, consequences, timing, statistics, runtime, memory consumption, processor usage, and / or other metadata associated with the execution of the neural network to process the input. Once coverage and / or objectives are computed, the mutated input can be added to the corpus if it exercises new coverage, and / or added to the list of test cases if it causes the objective function to be satisfied.

[0046] For example, an objective function can be used to evaluate whether a certain state has been reached (e.g., an error state, etc.). The objective function can be applied to the metadata array, and inputs that result in the goal being met can be marked. The neural network debugging system can determine, based on the coverage array, whether the coverage provided by the mutated input is new coverage (e.g., whether the neural network has reached a state it had not previously reached, etc.). For example, in some embodiments, when a new activation vector is received, its nearest neighbor can be determined (e.g., by performing an approximate nearest neighbor algorithm) and how far away the nearest neighbor is (e.g., in units of Euclidean distance). If the distance is greater than some defined amount, then the input can be added to the corpus.

[0047] In some implementations, input mutation can be performed as a batch, and the batch of inputs can be fed into the computation graph. Coverage and objective functions can then be checked on a batch of output arrays.

[0048] Reference will now be made in detail to the embodiments, one or more examples of which are shown in the accompanying drawings. Each example is provided by way of explanation of the embodiments and does not limit the present disclosure. Indeed, it will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments without departing from the scope or spirit of the present disclosure. For example, features shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that all aspects of the present disclosure cover such modifications and variations.

[0049] Example devices and systems

[0050] Figure 1 A block diagram of an example computing system 100 providing for use of machine learning according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0051] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0052] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store instructions 118 and data 116 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0053] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0054] In some implementations, one or more machine learning models 120 can be received from a server computing system 130 via a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 can implement multiple parallel instances of a single machine learning model 120.

[0055] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning models 140 may be implemented by the server computing system 130 as part of a cloud-based service. Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0056] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0057] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store instructions 138 and data 136 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0058] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0059] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0060] In some embodiments, server computing system 130 may also include a neural network debugging system 142, such as described herein with respect to Figure 2 For example, the neural network debugging system 142 can provide for performing coverage-guided fuzz testing using a corpus of inputs, for example, to provide testing of the neural network, such as to discover errors that may occur for rare inputs.

[0061] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0062] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operably connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store instructions 158 and data 156 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0063] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include truncated back propagation through time. The model trainer 160 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained. In particular, the model trainer 160 may train the machine learning models 120 and / or 140 based on a set of training data 162.

[0064] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0065] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard disk or optical or magnetic media.

[0066] Network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications through network 180 can be carried via any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0067] Figure 1 The diagram illustrates an example computing system that can be used to implement the present disclosure. Other computing systems may also be used. For example, in some embodiments, the user computing device 102 may include a model trainer 160 and training data 162. In such embodiments, the model 120 may be trained and used locally at the user computing device 102. In some such embodiments, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0068] Example Debug System Layout

[0069] Figure 2 A block diagram depicts an example neural network debugging system 200 using coverage-guided fuzz testing, according to an example embodiment of the present disclosure. In some embodiments, the neural network debugging system 200 can provide for performing coverage-guided fuzz testing using a corpus of inputs, for example, to test the neural network, such as to discover errors that may occur on rare inputs. The neural network debugging system 200 can allow for mutation of the corpus inputs guided by coverage metrics (e.g., randomly varying the inputs according to a mutation procedure and adding the mutated inputs to the input corpus when they exercise new coverage) toward the goal of satisfying user-specified constraints. As an example, coverage can be measured by analyzing activation vectors of a neural network coverage graph. For example, in some embodiments, new coverage can be determined based on whether the neural network has led to a state that the neural network had not previously reached, such that the new coverage helps provide incremental progress in debugging the neural network model. For example, in some embodiments, coverage-guided fuzz testing can be applied to neural networks to discover numerical errors in trained neural networks, generate discrepancies between neural networks and quantized versions of those networks, reveal unexpected behavior in the model, and the like.

[0070] like Figure 2As shown in FIG, a neural network debugging system 200 may include a coverage-guided fuzz tester 202 and a seed corpus 220 (e.g., comprising at least one set of inputs for a computation graph), which may provide an initial set of inputs to the coverage-guided fuzz tester 202 to test the neural network.

[0071] The coverage-guided fuzz tester 202 can obtain (e.g., select) a set of inputs from the seed corpus 220 to provide an input corpus 204, which can include all or a subset of the inputs included in the seed corpus 220. In some embodiments, the inputs can be restricted to a certain type of valid neural network input (e.g., images of the correct size and shape, characters from a vocabulary extracted from a training set, etc.). In some embodiments, the seed corpus 220 can be supplied by a user and / or can be selected from a set of available seed corpora. The inputs can be text inputs, image inputs, audio data inputs, sensor data inputs, and / or various other types of inputs.

[0072] The coverage-guided fuzz tester 202 may include an input selector 206 that may select input(s) from the input corpus 204 to use during a particular iteration of coverage-guided fuzz testing. For example, in some embodiments, the input selector 206 may be biased toward selecting inputs that were most recently added to the input corpus 204. As one example, the input selector 206 may select inputs using uniform random selection. For example, in some embodiments, the input selector 206 may use a random selection such as Heuristics to select input, where p(c k ,t) gives the selection of input corpus element c at time t k The probability of t k is element c k The time at which it was added to the input corpus. The intuition behind this is that recently sampled inputs are more likely to produce useful new coverage when mutated, but this advantage diminishes over time, so inputs can be selected as a function of their age.

[0073] The input selector 206 can provide the selected input(s) to the mutator 208. The mutator 208 can apply modifications (e.g., mutations) to the selected input(s) before providing the inputs to the neural network. For example, in some embodiments, the mutator 208 can add white noise with a user-configurable variance to the input(s) (e.g., image inputs, etc.). As another example, in some embodiments, the mutator 208 can add white noise with a user-configurable variance to the input(s) (e.g., image inputs, etc.), where the difference between the mutated inputs and the original inputs from which the mutated inputs originated is constrained to have a user-configurable L ∞ Norm. This type of constrained mutation can be useful for finding inputs that satisfy some objective function but are still reasonably of the same "class" as the original input used as the seed. In some embodiments, the image can be clipped after mutation to be in the same range as the input used to train the neural network being debugged.

[0074] As another example, in some embodiments, such as for text string input, one operation in a set of operations can be randomly and uniformly performed, including operations such as deleting characters at random positions, adding characters at random positions, replacing random characters at random positions, etc.

[0075] The input selector 206 includes computer logic for providing the desired functionality. The input selector 206 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the input selector 206 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the input selector 206 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard disk or optical or magnetic media.

[0076] The mutator 208 can then provide the mutated input(s) to the neural network 210. The neural network 210 can provide outputs that can include a set of coverage arrays for which coverage can be computed, and a set of metadata arrays from which the result of the objective function can be computed (e.g., data associated with a specific iteration associated with the provided input(s), etc.). For example, when the mutated inputs are fed into the computation graph, both the coverage arrays and the metadata arrays are returned as outputs.

[0077] Mutator 208 includes computer logic for providing the desired functionality. Mutator 208 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, mutator 208 comprises a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, mutator 208 comprises one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as a RAM disk, optical or magnetic media.

[0078] Objective function 212 can evaluate whether the neural network has reached a certain state, e.g., a state that can be considered an error, based on the metadata array(s). Error states can include incorrect predictions, execution times greater than a maximum execution time, processor usage greater than a maximum processor usage, neural network execution failures, and / or other instances of error or undesirable behavior or performance. In some embodiments, objective function 212 can be user-specified and / or selected from a set of available objective functions. Generally speaking, objective function 212 used to evaluate whether the neural network has reached a certain state can be separate and distinct from some other objective function or loss function used to train the neural network. If objective function 212 is satisfied, the mutated input(s) provided to the neural network can be marked, such as by adding them to a list of test cases (e.g., for future debugging, etc.). As an example, when mutated inputs are fed into a computation graph and a metadata array is returned as output, the objective function is applied to the metadata array and any mutated inputs that satisfy the objective function are marked.

[0079] The coverage analyzer 214 can determine, based on the coverage array(s), whether the coverage provided by the mutated input(s) is new coverage (e.g., whether the neural network has reached a state that was not previously reached, etc.). For example, in some embodiments, the coverage analyzer 214 can determine whether new coverage is provided based on whether the activation vector is approximately close to the previous activation vector. If the coverage analyzer 214 determines that the mutated input(s) provide new coverage, then the mutated input(s) can be added to the input corpus 204, for example, to be used as input(s) in future iterations of debugging, etc. For example, an approximate nearest neighbor can be calculated for the new activation vector and a check can be performed to determine how far the nearest neighbor is from the activation vector in Euclidean distance. If the distance is greater than a certain defined amount (e.g., which can be a user-configurable hyperparameter, an adaptive hyperparameter that adapts over time, and / or a dynamic hyperparameter that changes over time, for example, according to a predetermined schedule), then the input can be added to the corpus. In some implementations, the coverage-guided fuzz tester 202 may continue to select, mutate, and analyze inputs included in the input corpus 204 until instructed to stop and / or some other stopping criteria is met.

[0080] The coverage analyzer 214 includes computer logic for providing the desired functionality. The coverage analyzer 214 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the coverage analyzer 214 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the coverage analyzer 214 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard disk or optical or magnetic media.

[0081] The covering array and / or associated activation vectors may describe whether some or all neurons of the neural network were activated during processing of the input. As an example, the covering array and / or associated activation vectors may be restricted to describing whether only the logit of the neural network and / or neurons of a layer of the network prior to the logit were activated.

[0082] In some embodiments, the system 200 can be applied (e.g., in parallel) to two or more different (but potentially related) models to identify discrepancies between the models. For example, the two or more different models can be two or more different versions of a base model, such as a base model and a quantized version of the base model. To identify discrepancies, the same input (e.g., a mutated input) can be provided to the two or more different models, and two or more different outputs of the two or more different models can be analyzed (e.g., according to the objective function 212 and / or the coverage analyzer 214) to detect discrepancies or otherwise measure divergence in the outputs.

[0083] Example Method

[0084] Figure 3 Depicted is a flowchart of example operations for performing neural network debugging according to an example embodiment of the present disclosure. Although for purposes of illustration and discussion, Figure 3 The steps are depicted as being performed in a particular order, but the method of the present disclosure is not limited to the particular order or arrangement shown. The steps of method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0085] At 302, the computing system may obtain an input corpus, for example, from a seed corpus comprising one or more sets of inputs. For example, the seed corpus may contain at least one set of inputs for the computation graph. The inputs may be restricted to those that are, in some sense, valid neural network inputs. For example, if the inputs are images, the inputs may be restricted to those that are of the correct size and shape and are within the same interval as the input pixels of the dataset under consideration. As another example, if the inputs are sequences of characters, the inputs may be restricted to characters in a vocabulary extracted from a training set.

[0086] At 304, the computing system may select one or more inputs from the input corpus for use in debugging the neural network. For example, the computing system may select one or more inputs based on uniform random selection, based on one or more heuristics (e.g., Given a selection of input corpus element c at time t k The probability of t k is element c k The time at which the input is added to the input corpus, etc.) selects one or more inputs from the input corpus.

[0087] At 306, the computing system may modify the selected input(s) by performing some type of mutation on the selected input(s) before inputting into the neural network. For example, in some embodiments, the computing system may perform a simple modification of the input, such as flipping the sign of the input. As another example, in some embodiments, the computing system may restrict the modification to adhere to a constraint on the total modification of corpus elements over time.

[0088] At 308 , the computing system feeds the modified input(s) to the neural network to be debugged.

[0089] At 310, as a result of the neural network processing the one or more mutated inputs, the computing system can obtain a set of coverage arrays (e.g., describing whether one or more neurons of the neural network were activated during processing of the one or more mutated inputs by the neural network), which can be used to compute the actual coverage exercised by the modified input(s).

[0090] At 312, the computing system can determine whether the mutated input(s) provide new coverage, at least in part, on the coverage array(s). For example, if the neural network leads to a state it was not previously in, then the computing system can determine that new coverage is provided. If the mutated input(s) provide new coverage, then operations continue to 314. If the mutated input(s) do not provide new coverage, then operations continue to 322, where the next input can be analyzed. For example, in some embodiments, when a new activation vector is received, its nearest neighbor can be determined and checked to see how far away the nearest neighbor is in Euclidean distance. If the distance is greater than some defined amount, then the input can be added to the corpus.

[0091] At 314 , the computing system may add the mutated input(s) to the input corpus.

[0092] At 316 , as a result of the neural network processing the one or more mutated inputs, the computing system may extract a set of metadata arrays (e.g., describing metadata associated with the execution of the neural network processing the one or more mutated inputs) for use in computing the objective function.

[0093] At 318, the computing system may determine whether a target function is satisfied based at least in part on the metadata array(s). For example, the target function may evaluate whether the neural network has reached a particular state, such as a state that is considered an error. For example, the target function may be applied to the metadata array and inputs that cause the target to be satisfied may be marked. If the target function is satisfied, then the operation continues at 320. If the target function is not satisfied, then the operation continues at 322.

[0094] At 320 , the computing system can add the mutated input to the list of test cases.

[0095] Additional Disclosure

[0096] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0097] Although the present invention has been described in detail with respect to various specific example embodiments of the present invention, each example is provided by way of explanation, rather than by way of limitation. Those skilled in the art can easily produce variations, changes, and equivalents of such embodiments after obtaining an understanding of the foregoing. Thus, the present invention discloses and does not exclude such modifications, changes, and / or additions to the present invention, as will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that the present invention cover such variations, changes, and equivalents.

Claims

1. A computer-implemented method for debugging a neural network, the method comprising: Obtaining, by one or more computing devices, one or more inputs from an input corpus; mutating the one or more inputs by the one or more computing devices; providing one or more mutated inputs to the neural network by the one or more computing devices; obtaining, by the one or more computing devices as a result of the neural network processing the one or more varied inputs, a set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more varied inputs by the neural network; determining, by the one or more computing devices based at least in part on a set of covering arrays, whether the one or more mutated inputs provide new coverage; as well as After determining that the one or more mutated inputs provide new coverage, adding, by the one or more computing devices, the one or more mutated inputs to an input corpus; wherein the input is one of an image, text, audio, web page or document; and Wherein, determining, by the one or more computing devices based at least in part on the set of covering arrays, whether the one or more mutated inputs provide new coverage comprises: generating, by the one or more computing devices, an activation vector based at least in part on the set of covering arrays; executing, by the one or more computing devices, an approximate nearest neighbor algorithm to identify a previous activation vector; determining, by the one or more computing devices, a distance between the activation vector and a previous activation vector identified by an approximate nearest neighbor algorithm; and comparing, by the one or more computing devices, the distance to a threshold distance; When the distance is greater than a threshold distance, the one or more mutated inputs provide new coverage.

2. The method of claim 1, further comprising: obtaining, by the one or more computing devices as a result of the neural network processing the one or more mutated inputs, a set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determining, by the one or more computing devices, whether an objective function is satisfied based at least in part on the set of metadata arrays; as well as After determining that the objective function is satisfied, the one or more computing devices add the one or more mutated inputs to a list of test cases.

3. The method according to claim 1, wherein Obtaining one or more inputs from the input corpus includes selecting the one or more inputs from the input corpus using uniform random selection.

4. The method according to claim 1, wherein Obtaining one or more inputs from an input corpus involves using The heuristic is to select the one or more inputs from the input corpus, where p(c k ,t) gives the selection of input corpus element c at time t k The probability of t k is element c k The time at which the sentence was added to the input corpus.

5. The method according to claim 1, wherein Mutating the one or more inputs includes adding white noise having a user-configurable variance to the one or more inputs.

6. The method of claim 1, wherein: Mutating the one or more inputs includes adding white noise having a user-configurable variance to the one or more inputs, wherein the difference between the mutated inputs and the original inputs from which the mutated inputs originated is constrained to have a user-configurable L ∞ norm.

7. The method of claim 1, wherein: Determining whether the one or more mutated inputs provide new coverage includes determining whether the neural network has reached a new state that it has not previously reached.

8. The method of claim 7, wherein: Determining whether the neural network has reached a new state that it has not previously reached includes determining whether the activation vector is approximately close to a previous activation vector.

9. The method of claim 2, wherein: Determining whether the objective function is satisfied includes determining whether the neural network has reached a desired state.

10. The method of claim 9, wherein: The desired state is the error state of the neural network.

11. A computing device comprising: one or more processors; as well as one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing device to: Obtain one or more inputs from an input corpus; mutating the one or more inputs; providing one or more mutated inputs to the neural network; obtaining a set of covering arrays as a result of the neural network processing the one or more variant inputs, the set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more variant inputs by the neural network; determining whether the one or more mutated inputs provide new coverage based at least in part on the set of covering arrays; as well as After determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus; wherein the input is one of an image, text, audio, web page or document; and Wherein, determining whether the one or more mutated inputs provide new coverage based at least in part on the set of covering arrays comprises: generating an activation vector based at least in part on the set of covering arrays; Perform an approximate nearest neighbor algorithm to identify the previous activation vector; determining a distance between the activation vector and a previous activation vector identified by an approximate nearest neighbor algorithm; and comparing the distance to a threshold distance; When the distance is greater than a threshold distance, the one or more mutated inputs provide new coverage.

12. The computing device of claim 11 , further comprising instructions that, when executed, cause the computing device to: obtaining a set of metadata arrays as a result of the neural network processing the one or more mutated inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determining whether an objective function is satisfied based at least in part on the set of metadata arrays; as well as After determining that the objective function is satisfied, the one or more mutated inputs are added to a list of test cases.

13. The computing device of any one of claims 11 or 12, further comprising instructions that, when executed, cause the computing device to: An input corpus is obtained from a seed corpus, the seed corpus comprising at least one set of inputs.

14. The computing device of claim 11, wherein: Obtaining one or more inputs from the input corpus includes selecting the one or more inputs from the input corpus using uniform random selection.

15. The computing device of claim 11, wherein: Obtaining one or more inputs from an input corpus involves using The heuristic is to select the one or more inputs from the input corpus, where p(c k ,t) gives the selection of input corpus element c at time t k The probability of t k is element c k The time at which the sentence was added to the input corpus.

16. The computing device of claim 11, wherein: Mutating the one or more inputs includes adding white noise having a user-configurable variance to the one or more inputs.

17. The computing device of claim 11, wherein: Mutating the one or more inputs includes adding white noise having a user-configurable variance to the one or more inputs, wherein the difference between the mutated inputs and the original inputs from which the mutated inputs originated is constrained to have a user-configurable L inf norm.

18. The computing device of claim 11, wherein: Determining whether the one or more mutated inputs provide new coverage includes determining whether the neural network has reached a new state that it has not previously reached.

19. The computing device of claim 18, wherein: Determining whether the neural network has reached a new state that it has not previously reached includes determining whether the activation vector is approximately close to a previous activation vector.

20. The computing device of claim 12, wherein: Determining whether the objective function is satisfied includes determining whether the neural network has reached a desired state.

21. The computing device of claim 20, wherein: The desired state is the error state of the neural network.

22. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations comprising: Obtain one or more inputs from an input corpus; mutating the one or more inputs; providing one or more mutated inputs to the neural network; obtaining a set of covering arrays as a result of the neural network processing the one or more variant inputs, the set of covering arrays describing whether one or more neurons of the neural network were activated during processing of the one or more variant inputs by the neural network; determining whether the one or more mutated inputs provide new coverage based at least in part on the set of covering arrays; as well as After determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus; wherein the input is one of an image, text, audio, web page or document; and Wherein, determining whether the one or more mutated inputs provide new coverage based at least in part on the set of covering arrays comprises: generating an activation vector based at least in part on the set of covering arrays; Perform an approximate nearest neighbor algorithm to identify the previous activation vector; determining a distance between the activation vector and a previous activation vector identified by an approximate nearest neighbor algorithm; and comparing the distance to a threshold distance; When the distance is greater than a threshold distance, the one or more mutated inputs provide new coverage.

23. The one or more non-transitory computer-readable media of claim 22, wherein: The operations further include: obtaining a set of metadata arrays as a result of the neural network processing the one or more mutated inputs, the set of metadata arrays describing metadata associated with execution of the neural network processing the one or more mutated inputs; determining whether an objective function is satisfied based at least in part on the set of metadata arrays; and After determining that the objective function is satisfied, the one or more mutated inputs are added to a list of test cases.

24. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to implement the method of any one of claims 1-10.

Images