Neural network layer-by-layer debugging

Abstract

Techniques are disclosed for debugging execution of a neural network on a target processor. A reference processor can generate a plurality of first reference tensors for a neural network. The neural network can be repeatedly reduced to produce a plurality of lengths. For each of the lengths, a compiler converts the neural network into first machine instructions that are executed by the target processor to generate a first device tensor, and a debugger program determines whether the first device tensor matches a first reference tensor. A shortest length is identified in which the first device tensor does not match the first reference tensor. Tensor outputs are enabled for a lower level intermediate representation of the shortest neural network, and the neural network is converted into second machine instructions that are executed by the target processor to generate a second device tensor.

Description

BACKGROUND

[0001] Debugging is an important step for improving the functionality of computing devices at the hardware or software level. As computing devices and the software running on them become more complex, debugging techniques have expanded to more methods for detecting so-called bugs, vulnerabilities, or defects. Some debugging techniques involve setting a breakpoint at some location within the compiled instructions so that the application program is stopped or paused in place while different variables and data structures associated with the application program can be inspected. Other techniques can make inaccessible data written to memory while the application program is executing so that the data can be inspected after the application program is completed. While such techniques are useful in simple systems, they are of little value for more complex hardware and software setups.

[0002] Artificial neural networks, referred to herein as “neural networks,” are computing systems that have an architecture based on biological neural networks. A neural network can be trained using training data to learn how to perform a certain task, such as identifying or classifying physical objects, activities, people, etc. from images or videos. A neural network, such as a deep neural network, can include multiple layers of processing nodes. Each processing node in a layer can perform computations on input data generated by processing nodes in a previous layer to generate output data. For example, a processing node can perform a set of arithmetic operations, such as multiplication and addition, to generate intermediate output, or perform post-processing operations on the intermediate output to generate a final output. An artificial neural network, such as a deep neural network, can include thousands or more processing nodes and millions or more parameters.

[0003] A neural network can be implemented using a computer processor, such as a central processing unit (CPU), to perform computations. A neural network can also be implemented using a specialized processor optimized for parallel chain computations, which can include multiple processing units, such as a systolic array. In certain situations, such specialized processors can perform better than general-purpose processors for the same input data. BRIEF DESCRIPTION OF DRAWINGS

[0004] Various embodiments according to the present disclosure will be described with reference to the drawings, in which:

[0005] Figure 1 An example of a computational flow model for a neural network is shown.

[0006] Figure 2 An example set of reference tensors and device tensors generated for a neural network is shown.

[0007] Figures 3A-3D An example step of reducing the length of a neural network is shown.

[0008] Figure 4 An example step of debugging the execution of a shortened neural network is shown.

[0009] Figure 5 Example steps of debugging execution of a shortened neural network are shown.

[0010] Figure 6A and 6B An example of a method for debugging neural network execution on a target processor is shown.

[0011] Figure 7 A block diagram showing an example of a host system is included.

[0012] Figure 8 A block diagram showing an example of an acceleration engine is included.

[0013] Figure 9 A block diagram showing an example of an integrated circuit device is included.

[0014] Figure 10 A diagram of an example network is included. DETAILED DESCRIPTION

[0015] During system development, especially when two systems are fundamentally different, such as software and hardware, debugging the mismatch of functional behavior between the two systems can be a significant obstacle. In many cases, operations and computations dictated by a high-level software model can differ significantly from its hardware implementation.

[0016] Debugging execution of a neural network on a specialized processor, such as a processor having one or more systolic arrays, is challenging for a variety of reasons. One reason is that many intermediate results are not directly accessible by a debugger program. Because a systolic array can be composed of hundreds of individual data processing units (DPUs), it can be difficult to determine which DPU is at fault when multiple DPUs participate in the computation of a single output value. Moreover, when each DPU output cannot be directly inspected, it can be difficult to detect the location of precision-based summation errors, particularly for execution of neural networks that generally involve many parallel and sequential summations. Precision-based summation errors can occur when a sequence of finite-precision floating-point numbers is added. While these errors can be reduced in a model or reference computation by reordering the numbers from smallest to largest, the order of summation execution can not be changed during neural network execution and thus differs from the reference computation.

[0017] In addition to the overall result of a neural network, intermediate outputs of a neural network can also be used for debugging purposes. To access the intermediate outputs, the program code is typically modified and recompiled for execution. Re-compilation can consume a significant amount of time and resources, particularly for larger neural networks. Additionally, the modification of the program code can adversely affect the performance of the original program code, in some cases, changing the behavior of the neural network in a way that produces erroneous results.

[0018] Another challenge in debugging the execution of a neural network on a specialized processor involves the large number of computations to be collated. For example, for a single erroneous output value, there can be millions of computations performed on hundreds of layers and / or nodes of the neural network that contribute to the value. The size of the feature maps and the number of channels that can be employed increase the complexity of collating the intermediate results. Neural networks can include many convolution operations, which are complex operations that interleave dependencies of output values over a large set of input and intermediate values.

[0019] Examples described herein address these and other issues by providing techniques for debugging the execution of a neural network on a specialized processor, referred to herein as a target processor, using a layer-by-layer approach in which the neural network is first shortened in length to reduce the computational workload, and then "expanded" into multiple intermediate representations. The approach is advantageous because it allows tensor outputs at lower levels of the intermediate representations of the neural network to be available to a debugger program, whereas typically such tensors would be broken down, split, and / or merged with other tensors. The approach also minimizes any adverse impact due to adding additional instructions to the compiled code by focusing on a specific set of layers in the neural network that determine the origin of the error.

[0020] In some examples, a debugger program receives a request to debug the execution of a neural network on a target processor. The debugger program can simulate the execution of the neural network to generate reference tensors. The reference tensors can be based on sample inputs, such as feature maps. The sample inputs can be automatically generated by the debugger program, or can be provided by a user. In some examples, the reference tensors can be generated by a compiler during compilation.

[0021] After the neural network is compiled, the generated machine instructions are executed on the target processor to generate an output tensor. Comparing the output tensor to the reference output tensor reveals mismatches between the two. Next, the neural network is repeatedly shortened by reducing the number of layers of the neural network (e.g., from 100 to 99 layers, from 99 to 98 layers, etc.). Each shortened neural network is recompiled, and the generated machine instructions are re-executed by the target processor to generate new tensors ("device tensors"). Each of the device tensors is compared to the corresponding reference tensor to determine its accuracy and to determine whether the output of the shortened neural network failed. The iterative process continues until the shortest neural network whose output failed is identified.

[0022] The identified shortest (or in some examples, "shortened") neural network is then "expanded" by enabling tensor outputs of the neural network's lower level intermediate representations. This can occur during compilation by the compiler adding additional instructions to the machine instructions to cause the target processor to output device tensors of one or more of the lower level intermediate representations of the shortened neural network. These device tensors can be compared to the corresponding reference tensors to determine the last matching device tensor and the first non-matching device tensor. The debugger program can then identify the cause of the mismatch or calculation error based on the location of the last match and the first mismatch, the architecture of the target processor, and / or the particular intermediate representation in which the mismatch was found.

[0023] In the following description, various examples will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the examples. However, it will also be apparent to one skilled in the art that the examples can be practiced without the specific details. Furthermore, well-known features can be omitted or simplified in order not to obscure the embodiments being described.

[0024] Figure 1 An example of a computational flow model 100 for a neural network is shown. Neural networks take inspiration from the operational mechanisms of the human brain. According to various brain models, the main computational element of the brain is the neuron. Neurons are connected together with many elements, where the elements that enter a neuron are called dendrites, and the elements that leave a neuron are called axons. A neuron receives signals through the dendrites, performs computations on the signals, and outputs the signals on the axons. The input and output signals are called activations. The axon of one neuron can branch and connect to the dendrites of multiple neurons. The connections between the branches of the axon and the dendrites are called synapses.

[0025] Synapses scale the signals that pass through them. The scaling factors are called weights, and are thought to be the way in which the brain is able to learn: different weights for different reactions to inputs. Learning can change the weights, but the organization of the neurons and synapses does not need to change to learn content. The static structure of the brain can thus be used as a model for a program, and the weights can reflect the tasks that the program has learned to perform.

[0026] Neural networks operate based on the concept that the computation of a neuron involves a weighted sum of input values. These weighted sums correspond to the combination of value scaling performed by synapses and those values in a neuron. A function operation is performed on the combined inputs in a neuron. In brain models, the operation appears to be a non-linear function that causes a neuron to generate an output only if the input exceeds some threshold. Thus, by analogy, a node of a neural network can apply a non-linear function to the weighted sum of values input into the node.

[0027] In the illustrated example, the model 100 includes an input layer 104, intermediate layers often referred to as hidden layers 106, and an output layer 108. Each layer includes a number of nodes 102. In this example, the nodes 102 of the input layer 104 are connected to each node 102 of the hidden layer 106. The connections, which will be referred to as synapses in the brain model, are referred to as weights 110. Also in this example, each node 102 of the hidden layer 106 has a connection or weight 110 to each node 102 of the output layer. The input layer 104 can receive input and can propagate the input to the hidden layer 106. Neural network implementations can include multiple hidden layers. The weighted sum computed by the hidden layer 106 (or multiple hidden layers) is propagated to the output layer 108, which can present a final output to a user. In accordance with the brain model, the output of a node 102 can be referred to as an activation.

[0028] Examples of the computations that can occur at each layer in the example model 100 are as follows:

[0029]

[0030] In the above equations, W ij is a weight, x i is an input activation, y j is an output activation, f() is a non-linear function, and b is a bias term. Various non-linear functions can be used for different purposes.

[0031] The model 100 can be referred to as a directed weighted graph. In a directed graph, each connection to or from a node indicates a direction (e.g., into or away from the node). In a weighted graph, each connection can have a weight. For ease of understanding and debugging, tools used to develop neural networks can visualize the neural network as a directed weighted graph. In some cases, these tools can also be used to train the neural network and output trained weight values. Then, executing the neural network is a matter of using the weights to compute on input data.

[0032] Neural networks with many layers are able to learn more complex and more abstract high-level features than shallower networks. For example, a neural network can be taught to recognize images. In this example, the pixels of an image can be fed into the input layer of the neural network, and the output of the first layer can indicate the presence of low-level features such as lines and edges in the image. At subsequent layers, these features can be combined to measure the possible presence of higher-level features: lines can be combined into shapes, shapes can be further combined into collections of shapes. Given all this information, the neural network can output a probability that a high-level feature represents a particular object or scene. For example, the neural network can output whether an image contains a cat or not.

[0033] The learning phase of a neural network is referred to as training the neural network. During training, the neural network is taught to perform a task. In learning the task, values for the weights (and possibly also biases) are determined. The underlying program for the neural network (e.g., the organization of nodes into layers, the connections between nodes of each layer, and the computations performed by each node) need not change during training. After training, the neural network can perform the task by computing results using the weight values determined during training. For example, the neural network can output a probability that an image contains a particular object, can output a probability that an audio sequence contains a particular word, can generate a bounding box around an object in an image, or can suggest an action to take, etc. The program that runs the neural network is referred to as inference.

[0034] There are multiple ways in which the weights can be trained. One method is referred to as supervised learning. In supervised learning, all training samples are labeled so that inputting each training sample into the neural network produces a known result. Another method is referred to as unsupervised learning, in which the training samples are not labeled and training consists of finding structure in the data or finding clusters in the data. Semi-supervised learning is between supervised learning and unsupervised learning. In semi-supervised learning, a subset of the training data is labeled. The unlabeled data can be used to define cluster boundaries, and the labeled data can be used to label the clusters.

[0035] Neural networks have been used in a variety of applications, including, for example, in the fields of image and video, speech and language, medicine, gaming, and robotics. In image and video, neural networks have been used for image classification, object localization and detection, image segmentation, and action recognition. In speech and language, neural networks have been used for speech recognition, machine translation, natural language processing, and audio generation. In the medical field, neural networks have been used in genomics and medical imaging. In gaming, neural networks have been used to play video games and board games, including games with a large number of possible moves, such as Go. In robotics, neural networks have been used for motion planning for robots, visual navigation for robots, control stabilization, and driving strategies for autonomous vehicles.

[0036] Figure 2 A set of reference tensors 210 and device tensors 220 generated for a neural network 200 are shown in accordance with some examples of the present disclosure. The neural network 200 includes a plurality of layers 202, which are shown with simplified structures and nodes for illustrative purposes only. Each of the layers 202 can include one or more nodes that perform operations on input data to generate output data for the layer. A sample input 204 is provided as input to the neural network 200. The sample input 204 can be an image, a feature map, and / or one or more tensors. Each of the layers 202 can output one or more tensors 206 that are fed as input tensors into subsequent layers.

[0037] In some examples, the reference tensors 210 are generated by a reference processor based on the sample inputs 204. The operations associated with each of the layers 202 can be carried out on the reference processor, which has higher or similar precision than that provided by the target processor. Moreover, when generating the reference tensors 210, any modifications made to the operations by the compiler during compilation of the neural network 200 can be omitted. Thus, the reference tensors 210 provide a reference against which the device tensors 220 can be compared.

[0038] In some examples, the device tensors 220 are generated by compiling the neural network 200 into machine instructions and executing the machine instructions on the target processor. In some cases, the device tensors 220 corresponding to the total output of the neural network 200 are available, while intermediate device tensors overwritten during execution of the machine instructions are not available. After execution, the debugger program can compare the output device tensors to their corresponding reference tensors 212-11, as indicated by the comparison 230. Based on the comparison 230, the debugger program can determine that the output device tensors do not match the reference tensors 212-11 (i.e., T D ≠ T R ), for example, because the difference between the output device tensors and their corresponding reference tensors exceeds an accuracy threshold (e.g., 1%, 2%, 5%, 10%, etc.). Thus, the debugger program can determine that the output of the neural network 200 failed.

[0039] Figures 3A-3D Example steps to reduce the length of the neural network 200 are shown. Figure 3A A device tensor 320A generated for the shortened neural network 300A is shown, in accordance with some examples of the present disclosure. Figure 3A The reference tensors 210 are additionally shown. The shortened neural network 300A is produced by removing the last two layers of the neural network 200. The input to the shortened neural network 300A is the same sample input used for the neural network 200. The shortened neural network 300A is recompiled into machine instructions by the compiler, and the machine instructions are executed by the target processor using the sample input as input to generate a device tensor 320A that can only include the output device tensor. The debugger program compares the output device tensor to its corresponding reference tensor 212-9, as indicated by the comparison 330A. Based on the comparison 330A, the debugger program can determine that the output device tensor does not match the reference tensor 212-9 (i.e., T D ≠ T R ) because the difference between the output device tensor and the reference tensor 212-9 exceeds the accuracy threshold. Thus, the debugger program can determine that the output of the neural network 300A failed, and the neural network 300A will be shortened further.

[0040] Figure 3BAn apparatus tensor 320B generated for a shortened neural network 300B is shown in accordance with some examples of the present application. Figure 3B The reference tensors 210 are additionally shown. The shortened neural network 300B is produced by removing the first three layers of the shortened neural network 300A. The input to the shortened neural network 300B is the reference tensors 212-3. The shortened neural network 300B is recompiled by the compiler into machine instructions and executed by the target processor using the reference tensors 212-3 as input to generate an apparatus tensor 320B that can only contain the output apparatus tensor. The debugger program compares the output apparatus tensor to its corresponding reference tensor 212-9 as indicated by comparison 330B. Based on the comparison 330B, the debugger program can determine that the output apparatus tensor does not match the reference tensor 212-9 (i.e., T D ≠ T R ) because the difference between the output apparatus tensor and the reference tensor 212-9 exceeds the accuracy threshold. Thus, the debugger program can determine that the output of the neural network 300B failed and that the neural network 300B will be further shortened.

[0041] Figure 3C An apparatus tensor 320C generated for a shortened neural network 300C is shown in accordance with some examples of the present application. Figure 3C The reference tensors 210 are additionally shown. The shortened neural network 300C is produced by removing the last three layers of the shortened neural network 300B. The input to the shortened neural network 300C is the reference tensors 212-3. The shortened neural network 300C is recompiled by the compiler into machine instructions and executed by the target processor using the reference tensors 212-3 as input to generate an apparatus tensor 320C that can only contain the output apparatus tensor. The debugger program compares the output apparatus tensor to its corresponding reference tensor 212-6 as indicated by comparison 330C. Based on the comparison 330C, the debugger program can determine that the output apparatus tensor matches the reference tensor 212-6 (i.e., T D = T R ) because the difference between the output apparatus tensor and the reference tensor 212-6 does not exceed the accuracy threshold. Thus, the debugger program can determine that the output of the neural network 300C did not fail and that the length of the neural network 300C will be increased.

[0042] Figure 3D An apparatus tensor 320D generated for a shortened neural network 300D is shown in accordance with some examples of the present application. Figure 3DA reference tensor 210 is additionally shown. The shortened neural network 300B is produced by removing the last layer of the shortened neural network 300B (or by adding the last two layers back to the shortened neural network 300C). The input to the shortened neural network 300D is the reference tensor 212-3. The shortened neural network 300D is recompiled into machine instructions by the compiler, and the machine instructions are executed by the target processor using the reference tensor 212-3 as input to generate a device tensor 320D that can only contain the output device tensor. The debugger program compares the output device tensor to its corresponding reference tensor 212-8, as indicated by the comparison 330D. Based on the comparison 330D, the debugger program can determine that the output device tensor does not match the reference tensor 212-8 (i.e., T D ≠ T R ) because the difference between the output device tensor and the reference tensor 212-8 exceeds the accuracy threshold. Thus, the debugger program can determine that the output of the neural network 300D failed.

[0043] As shown with reference to Figures 3A-3D , the layers of the neural network are iteratively reduced until the neural network of the shortest length whose output device tensor does match its corresponding reference tensor is identified. The neural network can be shortened or lengthened by a variable amount in order to efficiently arrive at the shortest length. For example, a binary search can be employed, where the length is halved at each iteration. In some examples, the length is decremented by one at each iteration until the shortest length is identified. In some examples, the length is decremented by some amount based on the degree of mismatch with the reference tensor. For example, the length can be decremented by a larger amount when a larger mismatch is found, and the length can be decremented by a smaller amount when a smaller mismatch is found.

[0044] Figure 4 Example steps for debugging the execution of the shortened neural network 300D according to some examples of the disclosure are shown. In some examples, the reference tensors 410, 412, 414 are generated by a reference processor based on a sample input 404. The sample input 404 can be the same as the sample input 204 or the reference tensor 212-3, or in some examples, the sample input 404 can be a newly generated tensor. The operations associated with each layer of the shortened neural network 300D can be carried out on the reference processor to generate the reference tensor 410. Alternatively or additionally, the reference tensors 412, 414 can be generated for various intermediate representations 440 of the shortened neural network 300D. The intermediate representations 440 can include a high-level intermediate representation at which the reference tensor 410 is generated, and a first lower-level intermediate representation at which the reference tensor 412 is generated and a second lower-level intermediate representation at which the reference tensor 414 is generated.

[0045] Intermediate representations 440 can represent different representations that a neural network goes through when a compiler converts program code that is generally written using a high-level programming language into machine instructions that can be executed by a target processor. Lower-level intermediate representations can take into account the specific configuration of the target processor, and can thus contain additional details and additional operations, as compared to higher-level intermediate representations. In one example, converting from a high-level intermediate representation to a lower-level intermediate representation can include converting linear text representing a neural network into an intermediate graph structure that allows for flow analysis and operation reordering. Additional intermediate representations can include, but are not limited to: a parsed intermediate representation, a computation operation scheduler intermediate representation, a resource allocation intermediate representation, a scheduler intermediate representation, a code generator intermediate representation, a synchronization intermediate representation, machine code for one or more execution engines, runtime updates to machine code, and the like.

[0046] In some examples, when converting the shortened neural network 300D into machine instructions, the device tensors 420, 422, 424 are generated by enabling tensor outputs of the intermediate representations 440. In some examples, this is accomplished by the compiler inserting additional instructions into the machine instructions to cause the target processor to output tensors during execution. The additional instructions can include save operations. For example, a save operation can be introduced by inserting a node into a dataflow graph based on which the original program code was generated. After inserting these additional nodes, the dataflow graph can be converted into machine instructions in the same way that the dataflow graph would have been converted without the additional nodes being added.

[0047] After the shortened neural network 300D is compiled into machine instructions, the machine instructions are executed on the target processor, generating the device tensors 420, 422, 424. At each intermediate representation, a debugger program can compare the device tensors to their corresponding reference tensors to find the last matching tensor and the first non-matching tensor at each intermediate representation (“match-mismatch pair”). For example, the debugger program can identify the device tensor 423 of the first lower-level intermediate representation and the device tensor 425 of the second lower-level intermediate representation. The debugger program can inform a user of the match-mismatch pair found and the corresponding computations at each intermediate representation.

[0048] Figure 5Example steps for debugging execution of the shortened neural network 300D are shown in accordance with some examples of the present disclosure. In the example shown, tensor outputs are sequentially enabled for each of the intermediate representations 540 based on the match-mismatch pairs identified for the previous (higher level) intermediate representation (indicated by the dark colored rectangular boxes or containers). The shortened neural network 300D can be recompiled after a match-mismatch pair is identified for a particular intermediate representation, allowing tensor outputs to be enabled on a set of local operations directly related to the match-mismatch pair of the previous intermediate representation. Alternatively, as shown by the transition from device tensor 522 to device tensor 524, all tensors can be computed for the lower intermediate representation regardless of the match-mismatch pair identified. At each of the intermediate representations 540, corresponding reference tensors are generated for the device tensors, optionally also based on the match-mismatch pair identified.

[0049] In the example shown, tensor outputs are enabled for all operations during compilation, such that device tensor 520 is generated during execution of the compiled machine instructions. For the next lower level intermediate representation, tensor outputs are enabled for all operations during recompilation, such that device tensor 522 is generated during re-execution of the compiled machine instructions. For the next lower level intermediate representation, device tensor 524 is emulated by the reference processor since tensor outputs are not available. Device tensor 524 can be directly related to the match-mismatch pair identified in device tensor 522. As shown by the transition from device tensor 522 to device tensor 524, tensors can be decomposed into smaller sub-tensors (also referred to as tiling blocks) in the lower level intermediate representation. The lower level intermediate representation can also merge tensors, or in some rare cases, the tensors are slightly larger. For the next lower level intermediate representation, tensor outputs are only enabled for operations directly related to the match-mismatch pair identified in device tensor 524 during recompilation, such that device tensor 526 is generated during re-execution of the compiled machine instructions.

[0050] For the next lower level intermediate representation, tensor outputs are enabled during recompilation only for operations directly related to the match-mismatch pair identified in device tensor 526, such that device tensor 528 is generated during re-execution of the compiled machine instructions. For the next lower level intermediate representation, device tensor 530 is emulated by the reference processor because tensor outputs are not available. Device tensor 530 can be directly related to the match-mismatch pair identified in device tensor 528. For the next lower level intermediate representation, tensor outputs are enabled during recompilation only for operations directly related to the match-mismatch pair identified in device tensor 530, such that device tensor 532 is generated during re-execution of the compiled machine instructions. For the next lower level intermediate representation, tensor outputs are enabled during recompilation only for operations directly related to the match-mismatch pair identified in device tensor 532, such that device tensor 534 is generated during re-execution of the compiled machine instructions.

[0051] Figure 6A and 6B An example of a method 600 for debugging execution of a neural network on a target processor is shown. One or more steps of the method 600 can be performed in an order different than the example shown, and one or more steps of the method 600 can be omitted during performance of the method 600.

[0052] At step 602, a request to debug execution of a neural network on a target processor is received. In some examples, the request is received by a debugger program running on a host system. In some examples, the request indicates the neural network and / or the target processor. For example, the request can indicate where program code for the neural network is stored. In some examples, the neural network includes a plurality of layers. Each of the plurality of layers can include one or more operations.

[0053] At step 604, a plurality of first reference tensors for the neural network are generated based on a first sample input. In some examples, the plurality of first reference tensors are generated using a reference processor that is different than the target processor. In some examples, the plurality of first reference tensors are generated and / or received by the debugger program. For example, the debugger program can emulate execution of the neural network to generate the first reference tensors. In some examples, the first reference tensors are generated by a compiler during compilation.

[0054] At step 606, the plurality of layers of the neural network is iteratively reduced to produce a plurality of lengths of the neural network. For each particular length in the plurality of lengths, steps 608-612 are performed. The neural network can be shortened by a variable amount in order to efficiently arrive at the shortest neural network whose output fails. In some examples, each iteration shortens the neural network by a single layer. In some examples, each iteration shortens the neural network by two or more layers. To arrive at the shortest neural network whose output fails, a binary search can be employed, where each iteration halves the length. If a shortened neural network whose output does not fail is arrived at, the neural network can be lengthened by adding back the previously removed layers, as described with reference to 3A-3D.

[0055] At step 608, the neural network having the particular length is converted (or compiled) into first machine instructions. In some examples, the neural network having the particular length is converted by a compiler. In some examples, the compiler runs on a host system.

[0056] At step 610, the first machine instructions are executed using the target processor to generate a first device tensor. In some examples, the first machine instructions are executed based on a first sample input or based on one of the plurality of first reference tensors.

[0057] At step 612, it is determined whether the first device tensor matches a first reference tensor of the plurality of first reference tensors. The first reference tensor can be the reference tensor of the plurality of first reference tensors that corresponds to the first device tensor. In some examples, the debugger program determines whether the first device tensor matches the first reference tensor. In some examples, two tensors match if the difference between the two tensors does not exceed an accuracy threshold.

[0058] At step 614, a shortened length of the plurality of lengths in which the first device tensor does not match the first reference tensor is identified. In some examples, the shortest length of the plurality of lengths in which the first device tensor does not match the first reference tensor is identified. In some examples, step 614 is performed by the debugger program.

[0059] At step 616, a plurality of second reference tensors of a lower-level representation of the neural network is generated based on a second sample input. The second sample input can be different or the same as the first sample input. In some examples, the plurality of second reference tensors is generated using a reference processor. In some examples, the lower-level representation of the neural network is an intermediate representation of the neural network that is lower than a highest-level intermediate representation (e.g., program code) and higher than a lowest-level intermediate representation (e.g., machine instructions). In some examples, the plurality of second reference tensors is generated by the debugger program. For example, the debugger program can emulate execution of the neural network to generate the second reference tensors. In some examples, the second reference tensors are generated by the compiler during compilation.

[0060] At step 618, the neural network having the shortened length is converted (or compiled) into second machine instructions. In some examples, the neural network having the shortened length is converted by a compiler.

[0061] At step 620, a tensor output of the lower level representation is enabled when the second machine instructions are executed by the target processor. In some examples, during compilation, the compiler adds additional instructions to the second machine instructions that will cause the tensor output of the lower level representation. For example, one or more of the additional instructions can correspond to a save operation. In some examples, the additional instructions can further enable multiple tensor outputs of the lower level representation.

[0062] At step 622, the second machine instructions are executed using the target processor to generate a second device tensor of the lower level representation. In some examples, the second machine instructions are executed based on a second sample input or based on one of the multiple second reference tensors. For examples in which the additional instructions enable multiple tensor outputs of the lower level representation, executing the second machine instructions can cause a third device tensor of a second lower level representation of the neural network to be generated. In such examples, the lower level representation can be the first lower level representation.

[0063] At step 624, it is determined whether the second device tensor matches a second reference tensor of the multiple second reference tensors. The second reference tensor can be the reference tensor of the multiple second reference tensors that corresponds to the second device tensor. In some examples, the debugger program determines whether the second device tensor matches the second reference tensor. In some examples, two tensors match if the difference between the two tensors does not exceed an accuracy threshold. For examples in which the additional instructions enable tensor outputs of a second lower level representation of the neural network such that a third device tensor is generated, it can also be determined whether the third device tensor matches a third reference tensor of the multiple second reference tensors.

[0064] Figure 7 A block diagram showing an example of a host system 700 that can run, for example, the compiler 730 and the debugger 746 described herein is included. The host system 700 shown is an example of a computing device and includes a processor 702, a processor memory 704, at least one storage device 706, various input / output (I / O) devices 708, and at least one network interface 710. In Figure 7In the example of FIG. 7, host system 700 also includes an acceleration engine 712, which is an integrated circuit device that can accelerate certain operations or computations performed by host system 700. In various examples, host system 700 can be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smartphone, among other examples. In some examples, the operations or components discussed below as being performed or included in host system 700 can be performed or included in other computer devices. For example, compiler 730 can be executed on host system 700 while acceleration engine 712 is located at a different host system.

[0065] Processor 702 is an integrated circuit device that can execute program code in the form of instructions. The program code can be used by various software applications or tools, such as operating system 720, debugger 746, or compiler 730. When processor 702 executes a program, the instructions for the program can be stored in processor memory 704. The instructions can also be stored elsewhere, such as on storage 706, and can be loaded into processor memory 704 as needed by processor 702. Processor 702 can also use processor memory 704 for temporary storage of other data used by processor 702 in its operations. In various examples, processor memory 704 is a volatile memory type, such as a random access memory type, although alternatively or additionally, a non-volatile memory type can be used for processor memory 704.

[0066] Storage 706 is an example of a device that can include non-volatile memory. For example, storage 706 can be a magnetic disk drive, solid state drive, or optical drive, among other examples. Storage 706 can also be non-transitory, such that program code and other data stored on storage 706 persist when storage 706 is not powered.

[0067] Storage 706 is one example of a peripheral device, which is a component that can be coupled to host system 700 to add functionality to host system 700. Other examples of peripheral devices include input / output devices 708 and network interface 710. Input / output devices 708 can include user input and output devices, such as a keyboard, mouse, touchscreen, microphone, display screen, speaker, printer, and scanner, among other examples. Network interface 710, which can be implemented using a network interface card, can provide access to one or more networks. Network interface 710 can include, for example, physical ports for connecting network cables and / or wireless antennas to communicate with WiFi and / or cellular networks. Network interface 710 can also be described as an I / O device.

[0068] The accelerator engine 712 is also another type of peripheral device or I / O device. The accelerator engine 712 is a device that is specifically built to perform certain operations that can be performed by the processor 702 but can be performed more quickly by the accelerator engine 712. For example, the accelerator engine 712 can be a neural network accelerator and thus can perform large-scale parallel computations of a neural network more efficiently than if the computations were performed by the processor 702. As another example, the accelerator engine 712 can be a graphics processing unit (GPU) and can be optimized to perform computations needed for graphics rendering. Other examples of devices that can be implemented by the accelerator engine 712 include a cryptography accelerator, a compression and decompression accelerator, a 3D accelerator, a regular expression accelerator, a security accelerator, and the like.

[0069] In various examples, the accelerator engine 712 can execute program code to perform certain operations. For example, when the accelerator engine 712 is a neural network accelerator, the accelerator engine 712 can be programmed to execute a particular neural network, such as a neural network that performs image recognition or a neural network that performs machine translation. As another example, to support execution of a neural network, the accelerator engine 712 can be programmed to perform operations such as copying data for the neural network from, for example, the processor memory 704 into the accelerator engine 712, copying input data for the neural network from the processor memory 704 into the accelerator engine 712, and / or copying results from the accelerator engine 712 into the processor memory 704, among other examples.

[0070] To generate program code for the accelerator engine 712, in various examples, the host system 700 can execute a compiler 730. Generally, a compiler is a software program that translates program code written in a human-readable language into a format that can be read and processed by an integrated circuit device, such as machine instructions. In Figure 7 In examples in which the accelerator engine 712 is a neural network accelerator, and the compiler 730 is used to compile a neural network description into instructions to be executed by the accelerator engine 712. Another compiler can be used when the accelerator engine 712 implements a different type of accelerator.

[0071] The compiler 730 can be activated, for example, when the operating system 720 receives a keyboard, mouse, touchscreen, voice command, or other input from the input / output devices 708. The input can also include input code 742 for the compiler 730, such as options to compile and configure the compilation process. After the compiler 730 is activated, the processor 702 can load instructions for the compiler 730 into the processor memory 704 and can execute the instructions.

[0072] In Figure 7In the example of FIG. 7, the compiler 730 includes a first stage 732, a second stage 736, and a third stage 740 that each perform different operations to produce the compiled code 744. In other examples, the compiler 730 can combine the operations of the first stage 732, the second stage 736, and / or the third stage 740 into fewer stages, or can divide the operations of one of the stages into multiple stages.

[0073] The first stage 732 can receive and process the input code 742. The input code 742 can describe a program in a high-level programming language such as Java, C++, or Tensorflow, among many other examples. The input code 742 can describe steps that perform, for example, image recognition, speech recognition, machine translation, or other operations. The input code 742 can be obtained, for example, from the storage 706. Alternatively, although not shown here, the input code 742 can be located in the processor memory 704 or can be obtained from a network location using the network interface 710. The processing of the input code 742 can include ordering the operations described in the input code 742 into layers, where the output of one layer provides input to the next layer. The processing can also include identifying steps to be performed by the processor 702, rather than by the acceleration engine 712. For example, through execution of the driver 722, the processor 702 can need to perform steps such as configuring direct memory access (DMA) descriptors for moving data into or out of the acceleration engine 712, among other examples.

[0074] The output 734 of the first stage 732 can be organized, for example, in layers, nodes, and connections between nodes of a neural network. The second stage 736 can perform intermediate processing on this output 734. For example, the operations performed in any one layer or at any one node in a layer can be too much for the acceleration engine 712 to perform at once. The acceleration engine 712 can, for example, have a limited amount of local storage space for data needed for the computations, or the computations can exceed the amount that the acceleration engine 712 can perform at once. In this example, the first stage 732 can break the operations of a layer or node into smaller operations that can fit in the local memory of the acceleration engine and / or that can fit in the amount of computations that the acceleration engine 712 can perform at once. The processing of the output 734 of the first stage 732 can include other steps such as scheduling or determining the order in which the acceleration engine 712 and / or the processor 702 will perform the operations, among other examples.

[0075] In various examples, the output 738 of the second stage 736 includes various steps to be performed by components of the acceleration engine 712 in the order of the steps of execution. The output 738 can be represented, for example, as a dataflow graph, where the nodes in the graph represent memory operations, computations, and other operations, and the edges or connections between the nodes represent dependencies between the nodes, such as data dependencies, memory dependencies, or operation dependencies, among other examples.

[0076] The third stage 740 can operate on the output 738 of the second stage 736 and perform various steps before generating instructions to be executed by the acceleration engine 712. These steps may include, for example, removing redundant dependencies, resolving or disposing of dependencies between nodes by inserting synchronization instructions into the code, identifying possible optimizations in memory usage or memory bandwidth usage, and other operations.

[0077] The output of the third stage 740 is compiled code 744, which may contain machine instructions in binary format. In some examples, the compiled code 744 may be stored in processor memory 704. Alternatively or additionally, the compiled code 744 may be copied to storage device 706 or to a network location. As noted above, the acceleration engine 712 may be located at a different host system, in which case the compiled code 744 may be sent to another host system via network interface 710.

[0078] exist Figure 7 In the example, host system 700 may execute driver 722, which may also be referred to as device driver or runtime driver, to manage acceleration engine 712. Driver 722 provides an interface between an application running on host system 700 (or another host system) and acceleration engine 712. For example, driver 722 may provide an application programming interface (API) that defines functionality for feeding input data to acceleration engine 712 and defining operations to be performed on the input data. In this example and others, driver 722 may configure acceleration engine 712 to perform the operations. For example, driver 722 may identify the neural network that acceleration engine 712 will execute, and the location of compiled code 744 for the neural network in processor memory 704 or on storage device 706. Driver 722 may also load or cause acceleration engine 712 to load the compiled code 744, load or cause acceleration engine 712 to load input data that the neural network will operate on, and / or cause acceleration engine 712 to execute the input data. After acceleration engine 712 finishes, acceleration engine 712 can notify driver 722, and driver 722 can pass the result back to the application that requested the result.

[0079] In some embodiments, the host system 700 may execute a debugger 746, which is an application used to debug, inspect, and / or improve the functionality of a neural network. The debugger 746 allows a user to inspect the inputs and outputs of different layers and nodes of the neural network by setting breakpoints in compiled code 744. A breakpoint is a point where the executing code will stop further execution, where all execution state prior to the breakpoint is preserved. In various examples, breakpoints may be assigned to specific instructions, specific function calls, program counter values, or other sections of program code. Once a breakpoint is reached, the debugger allows the user to see the current values ​​of variables, the current contents of system memory, and / or step through the program code one or more instructions at a time, as well as other operations.

[0080] Figure 8 A block diagram illustrating an example of an acceleration engine 800 is included. The acceleration engine 800 is capable of containing components similar to... Figure 9 Examples of integrated circuits for one or more accelerators 802a-802n are shown in the figure.

[0081] exist Figure 8 In the examples, the acceleration engine 800 includes multiple accelerators 802a-802n, each of which can perform a set of operations. In various examples, accelerators 802a-802n are used for specific types of operations, such that accelerators 802a-802n can perform operations much faster than similar operations performed by a general-purpose processor. In various examples, in order to perform a set of operations, the input data for the operations to be performed must first be moved into accelerators 802a-802n. Additionally, in some cases, program code is also moved into accelerators 802a-802n, which programs the operations that accelerators 802a-802n will perform on the data. In the example shown, the acceleration engine 800 includes n accelerators 802a-802n. Examples of accelerators that can be included in the acceleration engine 800 include graphics accelerators, floating-point accelerators, neural network accelerators, etc. In various examples, accelerators 802a-802n may be the same (e.g., each of the accelerators is a graphics accelerator) or may be different (e.g., accelerators 802a-802n include a graphics accelerator, a floating-point accelerator, and a neural network accelerator).

[0082] The example acceleration engine 800 also includes DRAM controllers 842a-842k for communicating with external memory. In this example, external memory is implemented using DRAM 830. In the illustrated example, the acceleration engine 800 includes k DRAM controllers 842a-842k, each of which can communicate with a separate set of DRAM groups. In other examples, other types of RAM technology can be used for external memory. The DRAM controllers 842a-842k can also be referred to as memory controllers.

[0083] In various examples, input data and / or program code for the accelerators 802a-802n can be stored in the DRAM 830. Different programs can cause the accelerators 802a-802n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program can configure the neural network accelerator to perform speech recognition, while another program can configure the neural network accelerator to perform image recognition. In various examples, different accelerators 802a-802n can be programmed with different programs, such that each accelerator performs a different set of operations. In various examples, the processors 848a-848s can manage movement of program code from the DRAM 830 to the accelerators 802a-802n.

[0084] The example acceleration engine 800 also includes I / O controllers 844a-844p for communicating with I / O devices 832 in the system. The acceleration engine 800 can communicate with I / O devices through, for example, a processor bus. In some examples, a Peripheral Component Interconnect (PCI) and / or a variation of the PCI bus protocol can be used to implement the processor bus. The processor bus can connect the acceleration engine 800 to I / O devices, such as input and output devices, memory controllers, storage devices, and / or network interface cards, among others. In some examples, the I / O controllers 844-844p can enable the acceleration engine 800 to act as an I / O device for a host processor. For example, the acceleration engine 800 can be a recipient of input data from a host processor and commands indicating operations (e.g., particular computations or analyses) to be performed on the input data. In the illustrated example, the acceleration engine 800 includes P I / O controllers 844a-844p, each of which can include a separate root complex and can communicate with a separate set of I / O devices 832. In other examples, other standardized bus protocols, such as Ultra Path Interconnect (UPI), can be used for the host bus. In other examples, a proprietary bus protocol can be used.

[0085] Movement of data in the acceleration engine 800 can be managed by one or more processors 848a-848s, which can also be referred to as data management processors. In various examples, the processors 848a-848s can be configured to move data between the DRAM 830 and the accelerators 802a-802n, between the DRAM 830 and the I / O devices 832, and / or between the DRAM 830 and the I / O controllers 844a-844p. Figure 8In the example of FIG. 8, the processing units 848a-848s are incorporated into the device (e.g., on the same silicon die). In other examples, the processing units 848a-848s can be external to the accelerator engine 800 (e.g., on a different die and / or in a different package). In some examples, the processing units 848a-848s can manage movement of data from the I / O devices 832 to the accelerators 802a-802n or the DRAM 830. For example, input data can be located at the I / O devices 832 or in processor memory, and the processing units 848a-848s can move the input from the I / O devices 832 or processor memory into the accelerators or into the DRAM 830. As another example, program code for the accelerators 802a-802n can be located on the I / O devices 832 or in processor memory.

[0086] The example accelerator engine 800 also includes DMA engines 846a-846d, which can move data between the accelerators 802a-802n, the DRAM controllers 842a-842k, and the I / O controllers 844a-844p. In the illustrated example, the accelerator engine 800 includes d DMA engines 846a-846d. In some implementations, the DMA engines 846a-846d can be assigned to specific tasks, such as moving data from the DRAM controllers 842a-842d to the accelerators 802a-802n, or moving data between the I / O controllers 844a-844p and the accelerators 802a-802n. These tasks can be assigned, for example, by queuing descriptors with the DMA engines 846a-846d, where the descriptors identify the address of a block of data and the operation to be performed (e.g., a read or a write). For example, a descriptor can direct a DMA engine to instruct a DMA controller to read a block of data from the DRAM 830. As another example, a descriptor can instruct a DMA engine to write data read by a DMA controller to an accelerator. Other descriptors can be used to move data from an accelerator to the DRAM 830.

[0087] In various examples, each of the processing units 848a-848s can be responsible for managing data movement for a different accelerator. In some examples, a processing unit can manage data movement for more than one accelerator. Similarly, in various examples, each of the processing units 848a-848s can be assigned to one or more DMA engines 846a-846d. In these and other examples, the association between the processing units 848a-848s, the accelerators 802a-802n, and the DMA engines 846a-846d is determined by the program code each respective processing unit is executing.

[0088] In the example acceleration engine 800, various components can communicate over a chip interconnect 820. The chip interconnect 820 includes primarily the wiring used to route data between components of the acceleration engine 800. In some cases, the chip interconnect 820 can include a minimal amount of logic, such as multiplexers to control data direction, flip-flops and timing logic to handle clock domain crossings.

[0089] Figure 9 is a block diagram illustrating an example of an integrated circuit device. Figure 9 Examples of the 900 illustrate an accelerator 902. In various examples, for a set of input data (e.g., input data 950), the accelerator 902 can perform computations using the processing engine array 910, the activation engine 916, and / or the pooling engine 918. In some examples, the example accelerator 902 can be an integrated circuit component of a processor, such as a neural network processor. The processor can have other integrated circuit components, including additional accelerator engines.

[0090] In various implementations, the memory sub-system 904 can include multiple memory banks 914. In these implementations, each memory bank 914 can be independently accessed, which means that a read of one memory bank does not depend on a read of another memory bank. Similarly, a write to one memory bank does not affect or limit a write to a different memory bank. In some cases, each memory bank can be read and written to simultaneously. Various techniques can be used to have independently accessible memory banks 914. For example, each memory bank can be a physically independent memory component that has an address space that is separate and independent of the address space of each other memory bank. In this example, each memory bank can have at least one read channel and can have at least one separate write channel that can be used simultaneously. In these examples, the memory sub-system 904 can permit simultaneous access to the read or write channels of multiple memory banks. As another example, the memory sub-system 904 can include arbitration logic such that arbitration between outputs of, for example, multiple memory banks 914 can enable more than one memory bank’s output to be used. In these and other examples, each memory bank can operate independently of any other memory bank, although overall managed by the memory sub-system 904.

[0091] Making memory banks 914 independently accessible can increase the efficiency of accelerator 902. For example, values can be read simultaneously and provided to each row of processing engine array 910 so that the entire processing engine array 910 can be used in one clock cycle. As another example, memory banks 914 can be read while results computed by processing engine array 910 are being written to memory subsystem 904. In contrast, a single memory can only be able to service one read or write at a time. With a single memory, multiple clock cycles can be needed to read input data for each row of processing engine array 910, for example, before processing engine array 910 can begin.

[0092] In various embodiments, memory subsystem 904 can be configured to simultaneously service multiple clients, including processing engine array 910, activation engine 916, pooling engine 918, and any external clients that access memory subsystem 904 through communication structure 920. In some embodiments, being able to service multiple clients can mean that memory subsystem 904 has at least as many memory banks as clients. In some cases, each row of processing engine array 910 can count as a separate client. In some cases, each column of processing engine array 910 can output results so that each column can count as a separate write client. In some cases, outputs from processing engine array 910 can be written to memory banks 914, which can then provide input data to processing engine array 910. As another example, activation engine 916 and pooling engine 918 can include multiple execution lanes, each of which can be a separate memory client. For example, memory banks 914 can be implemented using static random access memory (SRAM).

[0093] In various embodiments, memory subsystem 904 can include control logic. For example, control logic can keep track of the address space of each of memory banks 914, identify which memory banks 914 to read or write, and / or move data between memory banks 914. In some embodiments, memory banks 914 can be hardwired to particular clients. For example, a set of memory banks 914 can be hardwired to provide values to rows of processing engine array 910, with one memory bank servicing one row. As another example, a set of memory banks can be hardwired to receive values from columns of processing engine array 910, with one memory bank receiving data for one column.

[0094] The processing engine array 910 is a computational matrix of the example accelerator 902. For example, the processing engine array 910 can perform parallel integration, convolution, correlation, and / or matrix multiplication, among other operations. The processing engine array 910 includes a plurality of processing engines 911 arranged in rows and columns, such that a result output by one processing engine 911 can be directly input into another processing engine 911. Thus, processing engines 911 that are not on the outer edges of the processing engine array 910 can receive data for operations from other processing engines 911 rather than from the memory subsystem 904.

[0095] In various examples, the processing engine array 910 uses systolic execution, in which data arrives at each processing engine 911 from different directions at regular intervals. In some examples, input data can flow in from the left into the processing engine array 910, and weight values can be loaded at the top. In some examples, weights and input data can flow from the left, while partial sums can flow from the top to the bottom. In these and other examples, the multiply-accumulate operations move through the processing engine array 910 as a diagonal wave front, with data moving right and down across the array. Control signals can be input on the left at the same time as the weights, and can flow through and down along with the computation.

[0096] In various implementations, the number of columns in the processing engine array 910 determines the amount of computation of the processing engine array 910, and the number of rows determines the memory bandwidth required to achieve maximum utilization of the processing engine array 910. The processing engine array 910 can have, for example, 64 columns and 428 rows, or some other number of columns and rows.

[0097] Figure 9 An example of a processing engine 911 is shown in the inset. As shown by this example, the processing engine 911 can include a multiplier-accumulator circuit. Input from the left can include, for example, input data i and a weight value w, where the input data is a value taken from a set of input data or a set of intermediate results, and the weight value is from a set of weight values that connect one layer of a neural network to the next layer. For example, a set of input data can be an image submitted for identification or object recognition, a piece of audio provided for speech recognition, a string of text for natural language processing or machine translation, or a current state of a game that needs to be analyzed to determine a next move, among others. In some examples, the input data and weight value are output to the right for input to the next processing engine 911.

[0098] In the illustrated example, input from above can include partial sum p_in provided from another processing engine 911 or from a previous round of computation by processing engine array 910. When computation starts for a new set of input data, the top row of processing engine array 910 can receive a fixed value for p_in, such as zero. As shown by this example, i and w are multiplied together, and the result is summed with p_in to produce a new partial sum p_out that can be input into another processing engine 911. Various other implementations of processing engine 911 are possible.

[0099] Output from the last row in processing engine array 910 can be temporarily stored in result buffer 912. The result can be an intermediate result that can be written to memory bank 914 for provision to processing engine array 910 for additional computation. Alternatively, the result can be a final result that, once written to memory bank 914, can be read from memory sub-system 904 through communication structure 920 for output by the system.

[0100] In some implementations, accelerator 902 includes an activation engine 916. In these implementations, activation engine 916 can combine results from processing engine array 910 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other examples, results from one or more columns in processing engine array 910 can need to be accumulated to produce an output activation for a single node in a neural network. In some examples, activation engine 916 can be bypassed.

[0101] In various examples, activation engine 916 can include multiple separate execution lanes. In these examples, the execution lanes can correspond to columns of processing engine array 910, and can perform operations on the output of a column, the results of which can be stored in memory sub-system 904. In these examples, activation engine 916 can be capable of performing between 1 and n parallel computations, where n is equal to the number of columns in processing engine array 910. In some cases, one or more of the computations can be performed simultaneously. Examples of computations that each execution lane can perform include exponential, square, square root, identity, binary step, bipolar step, sigmoidal, and ramp, among other examples.

[0102] In some embodiments, accelerator 902 can include a pooling engine 918. Pooling is a combination of the outputs of a column of processing engine array 910. The combination can include, for example, computing a maximum, a minimum, an average, a median, a sum, a multiplication, or another logical or mathematical combination. In various examples, pooling engine 918 can include a number of execution lanes that can operate on values from a corresponding column of processing engine array 910. In these examples, pooling engine 918 is capable of performing between 1 and n parallel computations, where n is equal to the number of columns in processing engine array 910. In various examples, the execution lanes of pooling engine 918 can operate in parallel and / or simultaneously. In some examples, pooling engine 918 can be bypassed.

[0103] Herein, activation engine 916 and pooling engine 918 can be referred to collectively as execution engines. Processing engine array 910 is another example of an execution engine. Another example of an execution engine is a direct memory access (DMA) engine that can be located outside of accelerator 902.

[0104] Input data 950 can arrive at communication fabric 920. Communication fabric 920 can connect accelerator 902 to other components of a processor, such as a DMA engine that can obtain input data 950 from an input / output (I / O) device, a storage drive, or a network interface. Input data 950 can be, for example, one-dimensional data, such as a string or a sequence of numerical values, or can be two-dimensional data, such as an array of pixel values for an image or frequency and amplitude values over time for an audio signal. In some examples, input data 950 can be three-dimensional, such as can be the case for situational information used by an autonomous vehicle or virtual reality data. In some embodiments, memory sub-system 904 can include a separate buffer for input data 950. In some embodiments, when accelerator 902 receives input data 950, input data 950 can be stored in memory bank 914.

[0105] In some examples, accelerator 902 can implement a neural network processing engine. In these examples, for a set of input data 950, accelerator 902 can run a neural network to perform a task, where the neural network has been trained for the task. Performing a neural network on a set of input data can be referred to as inference or performing inference.

[0106] The weights of the neural network can be stored in the memory sub-system 904 along with input data 950 that the neural network will operate on. The neural network can also include instructions that can program the processing engine array 910 to perform various calculations on the weights and input data. The instructions can also be stored in the memory sub-system 904, in the memory banks 914, or in a separate instruction buffer. The processing engine array 910 can output intermediate results representing the output of individual layers of the neural network. In some cases, activation engines 916 and / or pooling engines 918 can be enabled for calculations called by certain layers of the neural network. The accelerator 902 can store the intermediate results in the memory sub-system 904 for input into the processing engine array 910 to calculate the results of the next layer of the neural network. The processing engine array 910 can also output a final result from the last layer of the neural network. The final result can be stored in the memory sub-system 904, which is then copied out to a host processor memory or to another location.

[0107] Figure 10 An example network 1000 is included that can include one or more host systems, such as the host system shown in Figure 7 FIG. 1. For example, Figure 10 The example network 1000 includes a plurality of nodes 1002a-1002h, one or more of which can be, for example, a host system as shown in Figure 7 FIG. 1. Other ones of the nodes 1002a-1002h can be other computing devices, each of which includes at least a memory for storing program instructions, a processor for executing the instructions, and a network interface for connecting to the network 1000.

[0108] In various examples, the network 1000 can be used to process data. For example, input data can be received at one of the nodes 1002a-1002h or from other networks 1008 with which the network 1000 can communicate. In this example, the input data can be directed to a node in the network 1000 that includes an acceleration engine for the acceleration engine to operate on and produce a result. The result can then be communicated to other networks from which the input data was received. In various examples, input data can be accumulated from various sources, including one or more of the nodes 1002a-1002h and / or computing devices located in other networks 1008, and the accumulated input data can be directed to one or more host systems in the network 1000. Results from the host systems can then be distributed back to the sources from which the input data was collected.

[0109] In various examples, one or more of the nodes 1002a-1002h can be responsible for operations such as accumulating input data for host system operations, recording which host systems are busy and which host systems can accept more work, determining whether host systems are operating correctly and / or most efficiently, monitoring network security, and / or other management operations.

[0110] In Figure 10 In examples, the nodes 1002a-1002h are connected to each other using a switched architecture with point-to-point links. The switched architecture includes a plurality of switches 1004a-1004d, which can be arranged in a multi-layer network such as a Clos network. A network device that filters and forwards packets between segments of a local-area network (LAN) can be referred to as a switch. Switches typically operate at the data link layer (Layer 2) and sometimes at the network layer (Layer 3) of the Open Systems Interconnection (OSI) reference model, and can support several packet protocols. Figure 10 The switches 1004a-1004d can be connected to the nodes 1002a-1002h and provide multiple paths between any two nodes.

[0111] The network 1000 can also include one or more network devices, such as a router 1006, for connecting the network 1000 to other networks 1008. Routers use headers and forwarding tables to determine the best path for forwarding packets, and communicate with each other using protocols such as the Internet Control Message Protocol (ICMP) and the Border Gateway Protocol (BGP). Figure 10 The router 1006 can be used to connect to other networks 1008, such as subnets, LANs, wide-area networks (WANs), and / or the Internet.

[0112] In some examples, the network 1000 can include any one or combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and / or public networks. The interconnected switches 1004a-1004d and router 1006, if present, can be referred to as a switch fabric 1010, fabric, network fabric, or simply a network. In the context of computer networks, the terms "fabric" and "network" can be used interchangeably herein.

[0113] The nodes 1002a-1002h can be any combination of host systems, processor nodes, storage subsystems, and I / O chassis representing user devices, service provider computers, or third-party computers.

[0114] A user device can include a computing device that accesses an application 1032, such as a web browser or a mobile device application. In some aspects, the application 1032 can be hosted, managed, and / or provided by a computing resource service or service provider. The application 1032 can allow a user to interact with a service provider computer to, for example, access network content (e.g., web pages, music, videos, etc.). The user device can be a computing device such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a netbook computer, a desktop computer, a thin-client device, a tablet, an electronic book (e-book) reader, a game console, etc. In some examples, the user device can communicate with the service provider computer through other networks 1008. Additionally, the user device can be part of a distributed system managed, controlled by, or otherwise part of (e.g., a console device integrated with) the service provider computer.

[0115] Figure 10 Figure 10 The nodes of the network 1000 can also represent one or more service provider computers. The one or more service provider computers can provide native applications configured to run on user devices with which a user can interact. In some examples, the service provider computers can provide computing resources such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization, cloud-based software solutions, electronic content performance management, etc. The service provider computers can also operate to provide web page hosting, database, computer application development and / or implementation platforms, combinations of the foregoing, etc. to users. In some examples, the service provider computers can be provided as one or more virtual machines implemented in a hosted computing environment. The hosted computing environment can include one or more rapidly provisioned and released computing resources. These computing resources can include computing, networking, and / or storage devices. The hosted computing environment can also be referred to as a cloud computing environment. The service provider computers can include one or more servers, which can be arranged in a cluster, arranged as a server farm, or as individual servers not associated with one another, and can host the application 1032 and / or cloud-based software services. These servers can be configured as part of an integrated distributed computing environment. In some aspects, the service provider computers can additionally or alternatively include computing devices such as mobile phones, smart phones, personal digital assistants (PDAs), laptop computers, desktop computers, netbook computers, server computers, thin-client devices, tablet computers, game consoles, etc. In some cases, the service provider computers can communicate with one or more third party computers.

[0116] In one example configuration, the nodes 1002a-1002h can include at least one memory 1018 and one or more processing units (or processors) 1020. The processors 1020 can be implemented in hardware, computer-executable instructions, firmware, or combinations thereof. Computer-executable instruction or firmware implementations of the processors 1020 can include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

[0117] In some cases, the hardware processors 1020 can be single-core or multi-core processors. Multi-core processors can include multiple processing units within the same processor. In some examples, multi-core processors can share certain resources, such as a bus and second or third level caches. In some cases, each core in a single or multi-core processor can also include multiple execution logic processors (or execution threads). In such cores (e.g., cores with multiple logic processors), several stages of the execution pipeline and lower level caches can also be shared.

[0118] The memory 1018 can store program instructions that are loadable and executable on the processors 1020, as well as data generated during the execution of these programs. Depending on the configuration and type of the nodes 1002a-1002h, the memory 1018 can be volatile (such as RAM) and / or non-volatile (such as ROM, flash memory, etc.). The memory 1018 can include an operating system 1028, one or more data stores 1030, one or more application programs 1032, one or more drivers 1034, and / or services to implement the features disclosed herein.

[0119] The operating system 1028 can support basic functions of the nodes 1002a-1002h, such as scheduling tasks, executing applications, and / or controlling peripheral devices. In some embodiments, the service provider computer can host one or more virtual machines. In these embodiments, each virtual machine can be configured to execute its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, and the like. The operating system 1028 can also be a proprietary operating system.

[0120] Data store 1030 can include persistent or temporary data used and / or operated upon by operating system 1028, applications 1032, or drivers 1034. Examples of such data include web pages, video data, audio data, images, user data, and the like. In some embodiments, information in data store 1030 can be provided to user devices over network 1008. In some cases, data store 1030 can additionally or alternatively include stored applications and / or drivers. Alternatively or additionally, data store 1030 can store standard and / or proprietary software libraries, and / or standard and / or proprietary application program interface (API) libraries. Information stored in data store 1030 can be machine-readable object code, source code, interpreted code, or intermediate code.

[0121] Drivers 1034 include programs that can provide communication between components in a node. For example, some drivers 1034 can provide communication between operating system 1028 and additional storage 1022, network devices 1024, and / or I / O devices 1026. Alternatively or additionally, some drivers 1034 can provide communication between applications 1032 and operating system 1028 and / or between applications 1032 and peripheral devices accessible to the service provider computer. In many cases, drivers 1034 can include drivers that provide readily understood functionality (e.g., printer drivers, display drivers, hard drive drivers, solid state device drivers). In other cases, drivers 1034 can provide proprietary or specialized functionality.

[0122] The service provider computer or server can also include additional storage 1022, which can include removable storage and / or non-removable storage. Additional storage 1022 can include magnetic storage, optical discs, solid-state discs, flash memory, and / or tape storage. Additional storage 1022 can be housed in the same chassis as nodes 1002a-1002h, or can be in an external housing. Memory 1018 and / or additional storage 1022, and their associated computer-readable media, can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computing devices. In some embodiments, memory 1018 can include a variety of different types of memory, such as SRAM, DRAM, or ROM.

[0123] The removable and non-removable storage 1018 and the additional storage 1022 are examples of computer-readable storage media. For example, computer-readable storage media can include volatile or non-volatile, removable or non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. The memory 1018 and the additional storage 1022 are examples of computer storage media. Additional types of computer storage media that can be present in the nodes 1002a-1002h include, but are not limited to, PRAM, SRAM, DRAM, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROMs, DVDs or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, solid state drives, or some other mass storage device technology which can be used to store desired information and which can be accessed by the nodes 1002a-1002h. Computer-readable media also includes combinations of any of the above, including of a plurality of a single type of media.

[0124] Alternatively, or in addition, the computer-readable communication media can include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave or other transport. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0125] The nodes 1002a-1002h can also include I / O devices 1026, such as keyboards, mice, pens, voice inputs, touch inputs, displays, speakers, printers, etc. The nodes 1002a-1002h can also include one or more communication channels 1036. The communication channels 1036 can provide a medium through which various components of the nodes 1002a-1002h can communicate. The communication channel or channels 1036 can be in the form of a bus, ring, switch fabric, or network.

[0126] The nodes 1002a-1002h can also contain network devices 1024 that allow the nodes 1002a-1002h to communicate with stored databases, another computing device or server, user terminals, and / or other devices on the network 1000.

[0127] In some embodiments, network device 1024 is a peripheral device, such as a PCI-based device. In these embodiments, network device 1024 includes a PCI interface for communicating with a host device. The term "PCI" or "PCI-based" can be used to describe any of the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and Peripheral Component Interconnect Express (PCIe), or any other improvement or derivative protocol based on the PCI protocols discussed herein. PCI-based protocols are standard bus protocols for connecting devices, such as local peripheral devices, to host devices. A standard bus protocol is a data transfer protocol whose specifications have been defined and adopted by various manufacturers. Manufacturers ensure that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, a PCI-based device also includes devices that use Non-Volatile Memory Express (NVMe) communications. NVMe is a device interface specification for accessing non-volatile storage media attached to a computing system using PCIe. For example, a bus interface module can implement NVMe, and network device 1024 can connect to a computing system using a PCIe interface.

[0128] PCI-based devices can include one or more functions. A "function" describes an operation that can be provided by network device 1024. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption and decryption controllers, among others. In some cases, a PCI-based device can include more than one function. For example, a PCI-based device can provide a mass storage controller and a network adapter. As another example, a PCI-based device can provide two storage controllers to control two different storage resources. In some embodiments, a PCI-based device can have up to eight functions.

[0129] In some embodiments, network device 1024 can include Single-Root Input / Output Virtualization (SR-IOV). SR-IOV is an extended capability that can be included in PCI-based devices. SR-IOV allows a physical resource (e.g., a single network interface controller) to appear as multiple resources (e.g., sixty-four network interface controllers). Thus, for devices that utilize PCI-based devices, a PCI-based device that provides a certain functionality (e.g., a network interface controller) can appear to be multiple devices that provide the same functionality. The functionality of a SR-IOV capable storage adapter device can be classified as a physical function (PF) or a virtual function (VF). A physical function is a fully-featured function of a device that can be discovered, managed, and manipulated. A physical function has configuration resources that can be used to configure or control the storage adapter device. A physical function includes the same configuration address space and memory address space as a non-virtualized device. A physical function can have a number of virtual functions associated with it. A virtual function is similar to a physical function, but is a lightweight function that typically can lack configuration resources and is typically controlled by the configuration of its underlying physical function. Each physical function and / or virtual function can be assigned to a respective execution thread (e.g., a virtual machine) running on a host device.

[0130] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If the modules are software modules, the modules can be embodied on a non-transitory computer readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the described processes and architectures can be performed in real-time or in an asynchronous mode prior to any user interaction. The modules can be configured in the manner recommended in the preceding figures, and / or the functionality described herein can be provided by one or more modules existing as separate modules, and / or the module functionality described herein can be spread over multiple modules.

[0131] Accordingly, the specification and figures are to be regarded in an illustrative manner, and not a restrictive one. It is understood that the disclosure is intended to cover all modifications and alternative methods falling within the spirit and scope of the disclosure as set forth in the appended claims.

[0132] Other variations are within the spirit of the present disclosure. Therefore, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof have been shown in the drawings and described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims.

[0133] The use of the words "a" and "said" and similar referents in the context of describing the disclosed examples (especially in the context of the following claims) are to be understood as referring to one or more than one (i.e., to the singular, the plural, the equivalent of the plural, or the like). The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" is to be construed as partly or wholly inclosed, attached, or joined together, even if there is an intermediate material or some other item between the two parts or elements being connected. The recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated in the specification as if it were individually recited herein. The use of the term "about" to modify the recitation of a value, unless otherwise stated in this document, is intended to indicate that the value is within a reasonable range of the recited value, such as within 10% of the recited value. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate examples of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0134] Unless specifically stated otherwise, separable language such as "at least one of X, Y or Z," and the like, is intended to be understood as a separation of items X, Y, and Z into individual items, for example, X, Y, or Z, each being individually one of X, Y, or Z. Such separable language is generally intended to be understood to disclaim items such as "X, Y, and / or Z" along with any sub-set thereof unless otherwise specifically stated.

[0135] Various examples of the disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of these examples will become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the disclosure to be practiced with different and equivalent structures, materials, and methods than those specifically described herein. Accordingly, the disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the elements described above is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0136] As used below, any reference to a series of examples is to be understood as a separate reference to each of those examples (e.g., "Examples 1-4" is to be understood as "Example 1, 2, 3, or 4").

[0137] Example 1 is a method of debugging execution of a neural network on a target processor, the method comprising: receiving, by a debugger program running on a host system, a request to debug execution of a neural network on the target processor, the neural network comprising a plurality of layers; generating, using a reference processor on the host system and based on a first sample input, a plurality of first reference tensors of the neural network; iteratively reducing the plurality of layers of the neural network to produce a plurality of lengths, and for each particular length in the plurality of lengths: converting, by a compiler running on the host system, the neural network having the particular length into first machine instructions, executing the first machine instructions using the target processor and based on the first sample input or based on one of the plurality of first reference tensors to generate a first device tensor, and determining, by the debugger program, whether the first device tensor matches a first reference tensor of the plurality of first reference tensors; identifying a shortest length in the plurality of lengths in which the first device tensor does not match the first reference tensor; generating, using the reference processor and based on a second sample input, a plurality of second reference tensors of a lower-level representation of the neural network having the shortest length; converting, by the compiler, the neural network having the shortest length into second machine instructions, wherein the second machine instructions include additional instructions that enable tensor output of the lower-level representation; executing the second machine instructions using the target processor and based on the second sample input or based on one of the plurality of second reference tensors to generate a second device tensor of the lower-level representation; and determining, by the debugger program, whether the second device tensor matches a second reference tensor of the plurality of second reference tensors.

[0138] Example 2 is the method of example 1, wherein the additional instructions enable tensor output of a plurality of lower-level representations of the neural network.

[0139] Example 3 is the method of example 2, wherein executing the second machine instructions further generates a third device tensor of a second lower-level representation of the neural network, wherein the lower-level representation is a first lower-level representation.

[0140] Example 4 is the method of example 3, further comprising: determining, by the debugger program, whether the third device tensor matches a third reference tensor of the plurality of second reference tensors.

[0141] Example 5 is the method of examples 1-4, wherein the plurality of first reference tensors and the plurality of second reference tensors are generated by the debugger program.

[0142] Example 6 is a method of debugging execution of a neural network on a target processor, the method comprising: receiving a plurality of first reference tensors of a neural network; iteratively reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length of the plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions, executing the first machine instructions using the target processor to generate a first device tensor, and determining whether the first device tensor matches a first reference tensor of the plurality of first reference tensors; identifying a shortened length of the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors of a lower-level representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor of the lower-level representation.

[0143] Example 7 is the method of Example 6, wherein the shortened length is a shortest length of the plurality of lengths.

[0144] Example 8 is the method of Example 6 or 7, further comprising: determining, by the debugger program, whether the second device tensor matches a second reference tensor of the plurality of second reference tensors.

[0145] Example 9 is the method of any of Examples 6-8, wherein the second machine instructions include additional instructions that enable tensor outputs of the lower-level representation.

[0146] Example 10 is the method of Example 9, wherein the additional instructions enable tensor outputs of a plurality of lower-level representations of the neural network.

[0147] Example 11 is the method of Example 10, wherein executing the second machine instructions further generates a third device tensor of a second lower-level representation of the neural network, wherein the lower-level representation is a first lower-level representation.

[0148] Example 12 is the method of Example 11, further comprising: determining, by the debugger program, whether the third device tensor matches a third reference tensor of the plurality of second reference tensors.

[0149] Example 13 is the method of any of Examples 6-12, wherein the plurality of first reference tensors and the plurality of second reference tensors are generated by the debugger program.

[0150] Example 14 is a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: receiving a plurality of first reference tensors for a neural network; iteratively reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length in the plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions, executing the first machine instructions using a target processor to generate a first device tensor, and determining whether the first device tensor matches a first reference tensor in the plurality of first reference tensors; identifying a shortened length in the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors for a lower-level representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor for the lower-level representation.

[0151] Example 15 is the non-transitory computer-readable medium of Example 14, wherein the shortened length is a shortest length in the plurality of lengths.

[0152] Example 16 is the non-transitory computer-readable medium of Example 14 or 15, wherein the operations further include: determining, by the debugger program, whether the second device tensor matches a second reference tensor in the plurality of second reference tensors.

[0153] Example 17 is the non-transitory computer-readable medium of any of Examples 14-16, wherein the second machine instructions include additional instructions that enable tensor output for the lower-level representation.

[0154] Example 18 is the non-transitory computer-readable medium of Example 17, wherein the additional instructions enable tensor output for a plurality of lower-level representations of the neural network.

[0155] Example 19 is the non-transitory computer-readable medium of Example 18, wherein executing the second machine instructions further generates a third device tensor for a second lower-level representation of the neural network, wherein the lower-level representation is a first lower-level representation.

[0156] Example 20 is the non-transitory computer-readable medium of Example 19, wherein the operations further include: determining, by the debugger program, whether the third device tensor matches a third reference tensor in the plurality of second reference tensors.

[0157] Example 21 is a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of first reference tensors for a neural network; iteratively reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length in the plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions, executing the first machine instructions using a target processor to generate a first device tensor, and determining whether the first device tensor matches a first reference tensor in the plurality of first reference tensors; identifying a shortened length in the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors for a lower-level representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor for the lower-level representation.

[0158] Example 22 is the system of Example 21, wherein the shortened length is a shortest length in the plurality of lengths.

[0159] Example 23 is the system of Example 21 or 22, wherein the operations further comprise: determining, by the debugger program, whether the second device tensor matches a second reference tensor in the plurality of second reference tensors.

[0160] Example 24 is the system of any of Examples 21-23, wherein the second machine instructions include additional instructions that enable tensor output for the lower-level representation.

[0161] Example 25 is the system of Example 24, wherein the additional instructions enable tensor output for a plurality of lower-level representations of the neural network.

[0162] Example 26 is the system of Example 25, wherein executing the second machine instructions further generates a third device tensor for a second lower-level representation of the neural network, wherein the lower-level representation is a first lower-level representation.

[0163] Example 27 is the system of Example 26, wherein the operations further comprise: determining, by the debugger program, whether the third device tensor matches a third reference tensor in the plurality of second reference tensors.

Claims

1. A method of debugging execution of a neural network on a target processor, the method comprising: receiving a plurality of first reference tensors of a neural network; repeatedly reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length of the plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions; executing the first machine instructions using the target processor to generate a first device tensor; and determining whether the first device tensor matches a first reference tensor of the plurality of first reference tensors; identifying a shortened length of the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors of a representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor of the representation.

2. The method of claim 1, wherein the shortened length is a shortest length of the plurality of lengths.

3. The method of claim 1 or 2, further comprising: determining, by a debugger program, whether the second device tensor matches a second reference tensor of the plurality of second reference tensors.

4. The method of claim 1 or 2, wherein the second machine instructions include additional instructions that enable tensor outputs of the representation.

5. The method of claim 4, wherein the additional instructions enable tensor outputs of a plurality of representations of the neural network.

6. The method of claim 5, wherein executing the second machine instructions further generates a third device tensor of a second representation of the neural network, wherein the representation is a first representation.

7. The method of claim 6, further comprising: determining, by a debugger program, whether the third device tensor matches a third reference tensor of the plurality of second reference tensors.

8. The method of claim 1 or 2, wherein the plurality of first reference tensors and the plurality of second reference tensors are generated by a debugger program.

9. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of first reference tensors of a neural network; repeatedly reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length of the plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions; executing the first machine instructions using the target processor to generate a first device tensor; and determining whether the first device tensor matches a first reference tensor of the plurality of first reference tensors; identifying a shortened length of the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors of a representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor of the representation. executing the second machine instructions using a target processor to generate a second device tensor of the representation.

10. The non-transitory computer-readable medium of claim 9, wherein the shortened length is a shortest length of the plurality of lengths.

11. The non-transitory computer-readable medium of claim 9 or 10, wherein the operations further comprise: determining, by a debugger program, whether the second device tensor matches a second reference tensor of the plurality of second reference tensors.

12. The non-transitory computer-readable medium of claim 9 or 10, wherein the second machine instructions include additional instructions that enable tensor outputs of the representation.

13. The non-transitory computer-readable medium of claim 12, wherein the additional instructions enable tensor outputs of a plurality of representations of the neural network.

14. The non-transitory computer-readable medium of claim 13, wherein executing the second machine instructions further generates a third device tensor of a second representation of the neural network, wherein the representation is a first representation.

15. A system comprising: one or more processors; and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: receiving a plurality of first reference tensors of a neural network; repeatedly reducing a plurality of layers of the neural network to produce a plurality of lengths, and for each particular length of a plurality of lengths: converting, by a compiler, the neural network having the particular length into first machine instructions; executing the first machine instructions using a target processor to generate a first device tensor; and determining whether the first device tensor matches a first reference tensor of the plurality of first reference tensors; identifying a shortened length of the plurality of lengths in which the first device tensor does not match the first reference tensor; generating a plurality of second reference tensors of a representation of the neural network having the shortened length; converting, by the compiler, the neural network having the shortened length into second machine instructions; and executing the second machine instructions using the target processor to generate a second device tensor of the representation.

Images