Automatic debugging system and method for numerical error of deep learning compiler

Abstract

The invention discloses an automatic debugging system and method for deep learning compiler numerical errors, and the system comprises a model analysis module, a semantic matching module, a tracking module and a verification module, the analysis module receives and analyzes a defect model with compilation errors, carries out the extraction of the input of the defect model, sub-functions, and operators of each sub-function, and carries out the verification of the compilation errors. Constructing a symbolic calculation graph and an index table before and after optimization of the defect model; the semantic matching module performs hash processing on each node in the symbolic calculation graph, matches equivalent nodes before and after optimization by comparing the approximation degree of hash values between the nodes, and generates a matching relationship between the nodes and the equivalent nodes; the tracking module compares the data streams of the models before and after optimization according to the matching relationship, generates an error cumulant diagram by tracking the generation and propagation process of errors, and locates the modes causing the errors and compiler optimization transformation causing mode rewriting; and the inspection module inspects the correctness of the positioning result and finally outputs root information after inspection. According to the method, each computing node of the neural network model is quantified into a hash value capable of measuring the distance, compared with an existing positioning technology, the method can better adapt to the scene of a complex deep learning model, and compared with an incremental debugging technology, the error positioning speed and accuracy are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a technology in the field of deep learning, and specifically to an automated debugging system and method for numerical errors in deep learning compilers. Background Art

[0002] Deep learning compilers are a key technology in the field of deep learning, responsible for compiling, optimizing, and deploying neural network models. They compile neural network models into hardware-supported executable files, enabling the deployment of large, demanding interpreted deep learning models on devices with diverse resources and computing power. Furthermore, they leverage compile-time advantages to perform more complex optimizations on the model, accelerating model inference and reducing computing power requirements. Compared to traditional interpreted execution frameworks, compiler-optimized models can run dozens or even hundreds of times faster. This significant performance advantage has attracted widespread industry attention: AI companies in fields ranging from autonomous driving to facial recognition are using deep learning compilers to optimize their models and deploy them in real-world applications. However, while the compilation process delivers performance improvements, it also introduces numerical errors into the deep learning models themselves. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention proposes an automated debugging system and method for numerical errors in deep learning compilers, which quantifies each computing node of the neural network model into a hash value that can measure distance. Compared with existing positioning technologies, it is more adaptable to the scenarios of complex deep learning models, and significantly improves the error positioning speed and accuracy compared to incremental debugging technologies.

[0004] The present invention is achieved through the following undisclosed technical solutions:

[0005] The present invention relates to an automatic debugging system for numerical errors of a deep learning compiler, comprising: a model parsing module, a semantic matching module, a tracing module and a verification module, wherein: the parsing module receives and parses a defective model with compilation errors, extracts its input, sub-functions and operators of each sub-function, and constructs a symbolic computation graph and an index table before and after optimization of the defective model; the semantic matching module performs hash processing on each node in the symbolic computation graph, matches equivalent nodes before and after optimization by comparing the approximation of hash values ​​between nodes, and generates a matching relationship between the two; the tracing module compares data streams of the model before and after optimization based on the matching relationship, generates an error accumulation graph by tracing the generation and propagation process of the error, and locates the pattern that causes the error and the compiler optimization transformation that causes the pattern rewriting; the verification module verifies the correctness of the positioning result and finally outputs the root cause information after verification.

[0006] The automated debugging system further comprises a runtime control module, which supports the debugging device to complete the debugging work through the internal interface according to the compilation command, provides an external interface to support the user to set the tracking granularity, and generates a compilation signal to be sent to the compiler.

[0007] The debugging work includes: compiling and optimizing the model, running the model, and recording intermediate running data.

[0008] Setting the tracking granularity means controlling the granularity of error pattern tracking by setting an int value through an external interface. When the user sets the granularity to a small integer m, the maximum number of operators contained in each subfunction in the computation graph is m. This allows users to track the accumulation of errors more precisely.

[0009] The symbolic computation graph is a directed acyclic graph, in which each node represents a symbolic input or a symbolic sub-function of the model, and each directed edge represents the existence of a data dependency relationship between the nodes.

[0010] The symbolic input refers to: the input variable name, shape and data type of the defect model.

[0011] The symbolic sub-function refers to the abstract data structure of each sub-function of the model, including the operators contained in the sub-function, the calculation process, the number of input parameters and the output tensor shape.

[0012] The directed edge refers to the data flow relationship between sub-functions, and the direction pointed by the directed edge is the direction of data flow in the calculation graph.

[0013] The index table includes: N key-value pairs, where N is the number of computational graph nodes; each set of key-value pairs includes: the variable name or sub-function name of the node as the keyword part and the fingerprint of the node as the numerical part, which is a hexadecimal string with anti-hash collision characteristics. The index table corresponds one-to-one to the executable module actually corresponding to the symbolized node.

[0014] The matching relationship is obtained by performing semantic hashing on the symbolic computation graphs before and after optimization, that is, assigning a corresponding hash value to each node in the computation graph according to the semantics of the computation completed by the node, and then performing equivalent matching on all equivalent nodes in the computation graphs before and after optimization.

[0015] The hash value includes: the hash value of the input node in the calculation graph is 0; the hash value calculation process of the sub-function node is: summing the semantic codes of all operators contained in the sub-function node and adding the sum of the hash values ​​of all predecessor nodes of the sub-function node. When the hash value of the predecessor node has been referenced once in the hash value calculation process of other sub-functions, the hash value of the predecessor node is skipped in this addition.

[0016] The semantic encoding of operators means that operators with the same semantics are placed at the same level and share the same encoding. Operator encodings for operators with different semantics can vary significantly. Based on the encoding values, topologically different but equivalent operator combinations can be identified; unequal operator combinations are distinguished from each other and not confused.

[0017] The equivalent matching means that each node n1 in the optimized computation graph is traversed and compared with all nodes in the pre-optimization computation graph in turn. When the following conditions are met at the same time: the hash value distance between the two nodes is less than a threshold, and the shapes of the output tensors of the two nodes are the same, the two nodes are marked as a pair of matching nodes and stored in the equivalent node table.

[0018] The hash value being less than the threshold value means that the absolute value distance between the two hash values ​​is less than a very small limit.

[0019] The threshold is a*n+b, where n is the topological number of the node in the computational graph, 0 <a<1,b> 2.

[0020] The error occurs when, after a function is run in a computation graph, the computation results begin to differ, and the running results of the predecessor node of the sub-function are the same before and after optimization.

[0021] The generation and propagation of errors refer to the magnified or reduced numerical errors and their accumulation during the calculation process of the function nodes.

[0022] Tracing involves running both the optimized and unoptimized executable models, recording the output of each node in the computation graph. The module then compares the data flow errors of equivalent nodes using an equivalent node table. This involves calculating the computational error for each pair of equivalent nodes, starting with the smallest node in the topological order of the computation graph. After the computation is complete, information about the error generation and accumulation across the entire computation graph is obtained, allowing the generation and accumulation of numerical errors to be traced back to generate a cumulative error graph for each erroneous output node.

[0023] The error accumulation graph refers to a subgraph of the symbolic computation graph. Each end point in the error accumulation graph is an output node of the model with a numerical error. Each starting point in the graph is a node where the numerical error first occurs. The directed edges of the accumulation graph record the direction of error propagation, and each intermediate node stores the error information on the node.

[0024] The compiler optimization transformations for locating error-inducing patterns and causing pattern rewriting specifically include:

[0025] 1) Identify the nodes in the error accumulation graph where errors are generated or increased. Then, find the corresponding unoptimized equivalent node according to the node equivalence table, and compare the internal calculation processes of the two nodes. If the calculation process changes, the internal calculation process of the equivalent node is determined to be the mode that causes compilation errors (such as the sqrt-divide combination causing operator conversion); otherwise, the node has no error mode and only scales the error (such as the tan and relu operators).

[0026] 2) Use a fast binary search method to locate the erroneous optimization transformation: disable half of the suspicious transformations each time, that is, all transformations that perform optimization functions, recompile the erroneous pattern, and compare the recompiled running results with the running results of the pattern when the transformations are not disabled. This method is used to determine whether the transformation that caused the error is among the disabled transformations or the unselected transformations. Repeat the above binary search method until the erroneous optimization transformation is identified.

[0027] Verification involves disabling the incorrect optimization transformations resulting from positioning, completing other optimization transformations that are not disabled, and then comparing the error between the new model and the unoptimized model. If the new error decreases or disappears compared to the original, the verification module determines that the positioning is valid and outputs the positioning results and other debugging information, namely the error accumulation graph.

[0028] The present invention relates to an automated debugging method based on the above-mentioned system. After extracting the features of the defect model and constructing its calculation graph before and after optimization, each node in the graph is quantified into a hash value and node matching and operator calculation result comparison processing are performed to obtain the error source including the generation and propagation of the error, the pattern that causes the error, and the compiler optimization transformation that causes the pattern rewriting. Technical Effects

[0029] The static error localization algorithm based on hash matching in the present invention overcomes the difficulties faced by the open source community in locating numerical errors by symbolizing the neural network model, performing semantic-based hashing, matching equivalent nodes in the computational graph before and after optimization, generating an error propagation graph, and verifying the correctness of the localization. It can effectively diagnose various types of faulty neural networks with a detection rate of up to 100% and an accuracy rate of 100%. It can help artificial intelligence companies and R&D personnel automatically locate the root causes of numerical errors introduced by deep learning compilers, that is, the causes and locations of the errors; thereby saving researchers and engineers a lot of time and manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the system of the present invention;

[0031] Figure 2 Flowchart of the present invention;

[0032] Figure 3Schematic diagram of numerical error generation and propagation;

[0033] Figure 4 This is a schematic diagram of the core data structure obtained by parsing;

[0034] Figure 5 The symbolic computation graph obtained for the parsing module;

[0035] In the figure: the dotted box indicates the error accumulation diagram of the calculation graph: the error is generated during the calculation of sub-function 1 and propagated to sub-function 4;

[0036] Figure 6 This is a schematic diagram of the hash module working;

[0037] In the figure: the left figure shows the encoding of each operator predefined according to the hierarchical principle, and the right figure shows the hashing process of the node, and the encoding is obtained from the query on the left figure;

[0038] Figure 7 Schematic diagram of node matching constraints;

[0039] In the figure: When both constraints are satisfied, two nodes are equivalent;

[0040] Figure 8 This is the matching flow chart of the semantic matching module;

[0041] Figure 9 It is the workflow diagram of the tracking module;

[0042] Figure 10 The workflow diagram of the bit check module verification algorithm, in which x represents the input data of the model;

[0043] Figure 11 The effect of semantic matching and locating the root cause of the error in the embodiment;

[0044] In the figure: subgraph G1 is the original defect model; G2 and G3 are the unoptimized and optimized IRs of G1 respectively; G4 and G5 are the symbolic computation graphs corresponding to G2 and G3 respectively; G6 is the root cause of the error that was finally located. DETAILED DESCRIPTION

[0045] like Figure 1 As shown, this embodiment relates to an automated debugging system for numerical errors in deep learning compilers, including: a model parsing module, a semantic matching module, a tracing module, a verification module and a runtime control module, wherein: the runtime control module completes the compilation process of the model to be tested, that is: generating an intermediate representation from the model, optimizing the intermediate representation (IR) and translating it into executable code.

[0046] The described model parsing module includes: a parser and a storage unit, where: the parser parses the defect model to be debugged according to the principles of integrity, accuracy, and no side effects, and outputs the parsing results to two symbolic computation graphs P1 and P2.

[0047] The described symbolic computation graph and index table are recorded in the storage unit.

[0048] The principles of integrity, accuracy, and no side effects refer to: the information returned by the device must include the generation and propagation process of numerical errors, that is, integrity; and the device finally locates the operator mode rather than a single operator, that is, accuracy; and the device does not change or interfere with the model optimization process in the untracked mode, that is, no side effects.

[0049] The described semantic matching module includes: a hash table for recording semantic hash rules and a comparator, where: the comparator presets the logical judgment for determining whether two input nodes are equivalent. This module performs two stages of processing based on the symbolic computation graphs P1 and P2 obtained by the parsing module: the first stage performs semantic hashing on each symbolic computation graph; the second stage matches all equivalent nodes of the computation graphs before and after optimization, and all equivalent nodes are recorded in the equivalent node table as the output.

[0050] The described tracing module includes: a first error calculator and an error backtracking unit, where: the first error calculator receives the calculation results of two equivalent nodes, calculates their average relative error and records it on the corresponding nodes. When the errors of all equivalent nodes are calculated, an error table is obtained; the error backtracking unit locates the initial position where the error occurs by backtracking the error change process on the nodes of the computation graph according to the error table and the computation graph, and then determines the corresponding compiler optimization transformation that causes the error.

[0051] The described verification module includes: a second error calculator and a running interface, where: the running interface is connected to the DL compiler and passes the located error subgraph EM and error transformation as parameters to the DL compiler, and restarts the compilation of the DL compiler. During the compilation process, this error optimization transformation is disabled, and the compiler thus obtains a new executable model EM"; the compiler compiles EM again, and this time all optimization transformations are used. The executable model compiled the second time is EM'. The second error calculator calculates the error between the execution results of the subgraphs, that is, the error Err2 between EM"(x) and EM(x) and the error Err1 between EM'(x) and EM(x), where: x is the input of the error subgraph EM. When Err2 < Err1, the verification module determines that this positioning is valid. Otherwise, the positioning is invalid, and the verification module will continue to verify the correctness of other positionings.

[0052] The runtime control module includes: a compilation interface and a run command interface, wherein: the compilation interface transmits the compilation model, compilation level and specified disable / enable optimization transformation to the compiler; the run command interface starts the execution of the compiled model, and the run command interface receives the compilation / run command transmitted by the verification module. When the compilation command is transmitted, the compiler will compile the problem subgraph according to the specified disable / enable optimization transformation, thereby obtaining an executable model and returning it to the verification module; when the run command is transmitted, the compiler will execute the calculation process of the error subgraph EM / EM' / EM", and return the execution result EM(x) / EM'(x) / EM"(x) to the verification module.

[0053] The compiler performs optimization and analysis by modifying the optimized intermediate representation (IOR), ultimately generating a device-specific executable model based on the IOR. To diagnose numerical errors introduced by the deep learning compiler, a debugging mechanism examines the location of the error and how the original error affects the final output. This module then returns complete and useful trace information, stored as an error accumulation graph.

[0054] The computational graph means that any executable neural network model can be represented as a computational graph (,) where each vertex represents an input node or a sub-function processing unit, and the edge from A to B means that the data of A flows directly to B.

[0055] The error accumulation graph is a subgraph of the computation graph (,) and has multiple starting vertices and only terminal vertices. The terminal vertices represent the output nodes containing errors. Each source vertex represents the node that first generated an error, and the predecessor nodes of the source vertex are computationally accurate.

[0056] like Figure 3 As shown in FIG, the computation graph in this embodiment is a directed acyclic graph, and its error accumulation graph is represented by a dotted box. Figure 3 In the example, the computation in sub-function Func1 produces a numerical error, which propagates to sub-function Func4. The tracker will track the error through the backtracking mechanism and return an error accumulation graph for each erroneous output.

[0057] Given a model, error-triggering input, and compilation level, the debugging device establishes a correspondence between equivalent nodes in the optimized and unoptimized models, comparing the data flow of the model execution before and after optimization, thereby tracking the changes and occurrence of deep numerical errors. The entire process meets the principles of completeness, accuracy, and no side effects.

[0058] like Figure 2 As shown, this embodiment relates to an automated debugging method based on the above system, comprising the following steps:

[0059] Step 1) The model parsing module obtains the functional IR of the original defect model after optimization and before translation into machine code through the runtime control module. In the IR, the entire model expression is divided into multiple sub-functions, which serve as the smallest compilation unit. Each sub-function contains multiple parameters and expressions. An expression refers to an atomic calculation, such as =(,).

[0060] like Figure 2 As shown, the model parsing obtains the following data types: 1) Model input, i.e., variables and parameters passed to the model, which are recorded as separate nodes to track errors. 2) The number of occurrences of constants and temporary variables in a single sub-function, which helps to figure out how many times constant folding occurs during the optimization process. 3) Each sub-function, including the sub-function's constants and temporary variables, expressions, output tensor shapes, and signatures. Sub-functions are represented by independent nodes. Each node records the expressions it contains, the topological order of the final expressions, and the signature of the sub-function. The signature is the identification mark of the sub-function, which can retrieve the corresponding compilation module to obtain the specific output of the symbolic sub-function. The topological order of the expression indicates the number of calculations that the sub-function has completed. 4) Data flow, that is, recording the data flow relationship between the nodes of the symbolic computation graph to construct a symbolic computation graph. For example, when sub-function A consumes the global variable B, a directed edge from B to A will be added to the computation graph. The extracted nodes and data flow graph constitute the symbolic computation graph of the model. The composition of the symbolic computation graph is as follows Figure 3 As shown, the oval boxes indicate all inputs. Each subfunction includes multiple operators, and data flows between subfunctions. The endpoint of the data flow is the output node of the entire model. During a debugging session, the parsing module parses the optimized IR before and after optimization, generating two symbolic computation graphs: P1 and P2, the optimized and unoptimized computation graphs.

[0061] Step 2) The semantic matching module performs hash processing on each node in the symbolic computation graph, matches equivalent nodes before and after optimization by comparing the approximate degree of hash values ​​between nodes, and generates a matching relationship between the two. Specifically, it includes:

[0062] 2.1 Quantify the nodes of the symbolic computation graph into hash values ​​that can compare distances: For the symbolic computation graph, generate a hash value for each node in the graph, that is, the hash value represents the sum of the current and historical semantics of the subfunction, specifically: hash(a) = ∑ op∈a包含的算子 Encoding(op)+∑ n∈a的前驱节点 hash(n), where a is any subfunction.

[0063] For example, Figure 4The figure on the right shows the hash calculation process for the subfunction func2. The calculation sums the encodings of all operators contained in func2 and adds the hash value of func2's predecessor, func1. This hash function implicitly encodes positional information; nodes further back in the computation graph have higher hash values ​​than those at the front.

[0064] 2.2 Use hierarchical coding to distinguish operators: Hierarchical coding assigns the same code to operators with the same semantics, while assigning different codes to operators with different semantics. Based on the difference in code values, topologically different but equivalent operator combinations can be distinguished; at the same time, two unequal operator combinations are distinguished from each other and cannot be confused.

[0065] For example, the operator combination sqrt-divide and the operator combination rqsrt-multiple are both encoded as 0004, so they are equivalent. However, the encodings of conv-relu and multiple-relu are 0110 and 0012 respectively, which are not equivalent and will not be matched. Figure 4 The left figure shows the encoding of some operators. The encoding of conv2d is 0100, and this operator has the opportunity to be converted by the compiler into multiple other variants such as conv2d_transpose and sparse_conv2d. These equivalent operators essentially perform the same calculation as the dense operator. Therefore, conv and dense share the same encoding 0100. This value is different from the encoding 0002 of the multiplication operator. All ordinary binary operators, such as sqrt and divide, are encoded as 0002. For operators that do not perform actual operations but only change the shape and layout of tensors, they are encoded as 0000 because they do not introduce or change errors. Hierarchical encoding solves the matching difficulties caused by operator reordering.

[0066] During model optimization, multiple constant expressions are often simplified into expressions by the deep learning compiler, disrupting the semantic encoding-based matching process. The debugging device can solve this problem based on the characteristics of constant folding. The characteristic of constant folding is that for each folded expression, the number of parameters of the sub-function containing the expression will be reduced by 1. Based on this rule, the change in the number of temporary variables is taken into account when matching nodes, and the hash value is modified according to the increase or decrease in the number of variables, specifically: Where n is the topological order of the last expression in the function node, (n) is the distance control function, and p(a) computes the number of parameters contained in the function node. (n) is monotonically increasing, allowing for reasonable hash value variance. Figure 5The constraints for determining whether a node matches are: (1) the difference in hash values ​​between nodes 1 and 2 is less than a very small value, and (2) the output shapes of nodes 1 and 2 are the same. Only when both conditions are met is it considered a successful match; otherwise, the two nodes are not equivalent, i.e., node2 is not the node corresponding to node1 after optimization. Outshape refers to the output shape of the tensor, such as (2,5). Hash(const) means the encoding value of a constant, which is determined to be 2 in this invention. d(a,b) is the absolute value distance between the hash values ​​of two nodes a and b. When Match(a,b) = True, it means that a and b match successfully, otherwise the match fails.

[0067] 2.3 Matching process, such as Figure 6 As shown, specifically including:

[0068] 2.3.1: The hash module completes the hashing of each node in the symbolic computation graph before and after optimization and generates the hash value corresponding to the node.

[0069] 2.3.2: Set the initial value of the comparison position ba2.3.e to 0. This position records the position of the node in P2 that has not been compared. The setting of ba2.3.e can effectively reduce the number of comparisons and improve diagnostic efficiency.

[0070] 2.3.3: Determine whether there are any uncompared nodes in P1. If the condition is false, execute 2.3.11; if the condition is true, execute 2.3.4.

[0071] 2.3.4: Take out the node that has not been compared in P1 and record it as node1.

[0072] 2.3.5: Determine whether there are any uncompared nodes in P2. If not, execute 2.3.11. If there are any uncompared nodes, execute 2.3.6.

[0073] 2.3.6: Take the node with the subscript ba2.3.e from P2 and record it as node2.

[0074] 2.3.7: Obtain the hash values ​​of node1 and node2 and perform the conditional judgment described in Formula 2. If the hash values ​​satisfy Formula 2, execute 2.3.8; otherwise, execute 2.3.10.

[0075] 2.3.8: Combine the node names of node1 and node2 into key-value pairs and save them in the equivalent node table.

[0076] 2.3.9: Increase the value of ba2.3.e by 1 and execute 2.3.3 again.

[0077] 2.3.10: Find the successor node node3 of node2 in the computational graph P2, and determine whether the inequality d(node1, node2) < d(node1, node3) holds. If it holds, execute 2.3.3; if the inequality does not hold, execute 2.3.5.

[0078] 2.3.11: Output the equivalent node table.

[0079] Step 3) Trace the change process of the error through the tracing module, obtain the error accumulation graph, and locate the root cause of the numerical error, specifically including:

[0080] 3.1 Reproduce the error: The tracer reproduces the error calculation process by running the executable model again. For the optimized and unoptimized executable models, the tracer runs again in debug mode respectively. During the run, the debugging device collects the output of each sub-function. Through the signature of the sub-function, each specific output is mapped to a symbolic node. The tracer thus calculates the differences in the data flow of the optimized and unoptimized models on each equivalent node. During the calculation, the tracer calculates the calculation error between each node in P1 and its corresponding equivalent node in P2 along the data flow direction of the computational graph P1 in sequence. When there is no matching item for the node in the symbolic computational graph P1, the tracer skips that node. After the calculation is completed, the tracer obtains the error change situation of the optimized and unoptimized models.

[0081] 3.2 Trace back the numerical error: Trace back the numerical error along the reverse direction of the computational graph and generate an error accumulation graph. As Figure 7 shown, it is the overall tracing process. The tracer identifies all output nodes P1.outs of the optimized computational graph P1. Select the nodes with calculation errors from them. The tracer uses breadth-first search to find the error accumulation graph of each error node. The breadth-first algorithm is used to iteratively find the predecessor nodes where the current node has an error until all its predecessor nodes are found. The newly found nodes are added to the search list, and the next iteration will start from the first node in the search list. The nodes that have completed the search are removed from the search list. For the error accumulation graph, the termination condition of the search is that the search list is empty. The searched nodes are composed into an error accumulation graph according to their data flow order, reflecting the change process of the numerical error of the deep learning compiler along the computational graph.

[0082] 3.3 Locating Error Patterns and Related Optimization Transformations: Each error accumulation graph contains at least one sub-function that generates numerical errors, as well as multiple sub-functions that cause error scaling. When an error first occurs at node b in P1, the computation of b contains numerical errors. The tracking module finds the node b′ in P2 that corresponds to b. Based on the differences in their computational processes, the tracker can locate the pattern (operator combination) that causes the error. Furthermore, the debugging device uses a binary search method to locate the relevant transformations that cause the pattern rewrite. The tracker first identifies all transformations that perform optimization functions, which are all suspected optimization transformations. Next, it disables half of the suspected transformations at a time, recompiles the error pattern, and compares the new results with the results of the pattern when the transformations are not disabled. If the error disappears, the error-causing transformation is confirmed to be among the disabled transformations; otherwise, the target transformation is among the unselected transformations. The tracker can further narrow the scope of the target by repeatedly applying the binary search method described above until a unique transformation is identified. Because the transformation location process only compiles small operator combinations at a time, it takes very little time, negligible compared to the compilation time of the entire model.

[0083] like Figure 8 As shown in Figure 1, G1 is the user-entered defect model to be diagnosed. G2 and G3 represent the intermediate representation (IR) of the model before and after optimization, respectively. As indicated by the boldface in G2 and G3, the number of operators and the order of computation change in the IR before and after optimization, making data flow comparison difficult. G4 and G5 show the parsed symbolic computation graph, the hash values ​​of the nodes, and the node matching. As can be seen in the figure, even with changes in the order and number of operators, the debugging device still correctly matches two pairs of equivalent nodes. This is because the hash values ​​at the equivalent points are the same. A comparison of G4 and G5 reveals two pattern rewrites: the first is where the compiler rewrites sqrt-divide to rsqrt-multiply, with two multiplies folded; the second is where conv-multiply-negative-multiply-add is rewritten to multiply-negative-multiply-conv-add. Two related transformations were also identified.

[0084] Through specific practical experiments on Ubuntu 18.04.6LTS, when TVM version is v0.12.dev0., the results of running on 69 industrial models and synthetic models with different structures show that the proposed method can locate 87 error accumulation graphs and find 99 error root causes. Through manual analysis, the accuracy rate of these 99 root causes is 100%, which is higher than that of existing technologies (HUI GUO, IGNACIO LAGUNA, and CINDY 2020.pLiner: isolating lines of floating-point code for compiler-induced variability.[C]In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC'20).IEEE Press,Article 49,1–14) The method only found 9 errors, and only 8 of them were correct. The error localization speed of the present invention increased by 2.1-18.7 times compared with the comparison method.

[0085] Table 1 Comparison of positioning accuracy.

[0086] In the table, #Models represents the number of models for which the method can locate errors. The larger #Models, the more applicable the method is. Delta-debugging / delta is the abbreviation for the comparison method [1], and SEM is the abbreviation for the method of the present invention. RC is the number of error root causes found by the method, and Acc is the accuracy of the root causes, calculated by dividing the number of root causes correctly located by the method by all the root causes located.

[0087] Table 2: Positioning time of the comparative method and the method of the present invention.

[0088] P2-37 in the table represents the number of the deep learning model in the experiment, T SEM The time required to locate the root cause of the error for the present invention. delta The time required for the comparison method, the speedup ratio is T delta / T SEM .

[0089] Table 3 Complete coding table

[0090] Compared with the existing technology, the present invention can automatically debug more models, which is 7.67 times more than before; the accuracy of locating errors is higher; the error locating speed is greatly improved, and it can adapt to complex and large models and provide additional matching information. Therefore, the present invention can find all the changing processes of numerical errors, thereby outputting more complete and useful error information.

[0091] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. An automated debugging system for numerical errors in deep learning compilers, characterized by: include: Model parsing module, semantic matching module, tracing module and verification module, among which: the parsing module receives and parses the defective model with compilation errors, extracts its input, sub-functions and operators of each sub-function, and constructs the symbolic calculation graph and index table before and after the defective model is optimized; the semantic matching module hashes each node in the symbolic calculation graph, matches the equivalent nodes before and after optimization by comparing the approximate degree of hash values ​​between nodes, and generates a matching relationship between the two; the tracing module compares the data flow of the model before and after optimization according to the matching relationship, generates an error accumulation graph by tracing the generation and propagation process of the error, and locates the pattern that causes the error and the compiler optimization transformation that causes the pattern rewriting; the verification module verifies the correctness of the positioning result and finally outputs the root cause information after verification.

2. The automated debugging system for numerical errors of deep learning compilers according to claim 1, characterized in that: It further includes a runtime control module, which supports the debugging device to complete the debugging work through the internal interface according to the compilation command and provides an external interface to support the user to set the tracking granularity, and generates a compilation signal sent to the compiler; The debugging work includes: compiling and optimizing the model, running the model, and recording intermediate running data; The setting of tracking granularity refers to controlling the granularity of tracking error patterns by setting an int type number through an external interface. When the user sets the granularity to a smaller integer m, the maximum number of operators contained in each sub-function in the calculation graph is m, and the user can track the accumulation process of errors more finely.

3. The automated debugging system for numerical errors of a deep learning compiler according to claim 1 or 2, characterized in that: The symbolic computation graph is a directed acyclic graph, in which each node represents a symbolic input or a symbolic sub-function of the model, and each directed edge represents a data dependency relationship between nodes; The symbolic input refers to: the input variable name, shape and data type of the defect model; The symbolic sub-function refers to the abstract data structure of each sub-function of the model, including the operators, calculation process, number of input parameters and output tensor shape contained in the sub-function; The directed edge refers to the data flow relationship between sub-functions and the direction of the directed edge is the direction of data flow in the computation graph; The index table includes: N key-value pairs, where N is the number of computational graph nodes; each set of key-value pairs includes: the variable name or sub-function name of the node as the keyword part and the fingerprint of the node as the numerical part, which is a hexadecimal string with anti-hash collision characteristics. The index table corresponds one-to-one to the executable module actually corresponding to the symbolized node.

4. The automated debugging system for numerical errors of deep learning compilers according to claim 1, characterized in that: The matching relationship is obtained by performing semantic hashing on the symbolic computation graphs before and after optimization, that is, assigning a corresponding hash value to each node in the computation graph according to the semantics of the computation completed by the node, and then performing equivalent matching on all equivalent nodes in the computation graphs before and after optimization; The hash value includes: the hash value of the input node in the calculation graph is 0; the hash value calculation process of the sub-function node is: summing the semantic codes of all operators contained in the sub-function node and adding the sum of the hash values ​​of all predecessor nodes of the sub-function node. When the hash value of the predecessor node has been referenced once in the hash value calculation process of other sub-functions, the hash value of the predecessor node is skipped in this summing; The semantic encoding of operators means that operators with the same semantics are at the same level and share the same encoding, while operators with different semantics have significantly different encodings. Based on the encoding value, topologically different but equivalent operator combinations can be identified; two unequal operator combinations are distinguished from each other and will not be confused. The equivalent matching means that each node n1 in the optimized computation graph is traversed and compared with all nodes in the pre-optimization computation graph in turn. When the following conditions are met at the same time: the hash value distance between the two nodes is less than a threshold, and the shapes of the output tensors of the two nodes are the same, the two nodes are marked as a pair of matching nodes and stored in the equivalent node table.

5. The automated debugging system for numerical errors of deep learning compilers according to claim 4, characterized in that: The hash value being less than the threshold value means that the absolute value distance between the two hash values ​​is less than a very small limit; The threshold is a*n+b, where n is the topological number of the node in the computational graph, 0 <a<1,b> 2.

6. The automated debugging system for numerical errors of deep learning compilers according to claim 1, characterized in that: The error occurs when: after a function is run in the computation graph, the operation results begin to differ, and the operation results of the predecessor node of the sub-function before and after optimization are the same; The generation and propagation of errors refer to: the amplified or reduced numerical errors and their accumulation process during the calculation process of the function node; The tracking means that: the tracking module runs the optimized and unoptimized executable models respectively, and records the output of each node in the calculation graph during the running process; compares the data flow errors of the equivalent nodes according to the equivalent node table, that is, along the calculation order of the calculation graph, calculates the calculation error of each pair of equivalent nodes starting from the topologically smaller sequence number, and after the calculation is completed, obtains the error change information on the entire calculation graph, and then traces back the generation and accumulation process of numerical errors, and obtains an error accumulation graph for each erroneous output node.

7. The automated debugging system for numerical errors of a deep learning compiler according to claim 1, wherein: The error accumulation graph refers to a subgraph of the symbolic computation graph. Each end point in the error accumulation graph is an output node of the model with a numerical error. Each starting point in the graph is a node where the numerical error first occurs. The directed edges of the accumulation graph record the direction of error propagation, and each intermediate node stores the error information on the node.

8. The automated debugging system for numerical errors of a deep learning compiler according to claim 1 or 7, wherein: The compiler optimization transformations for locating error-inducing patterns and causing pattern rewriting specifically include: 1) Identify the nodes in the error accumulation graph where errors are generated or increased. Then, use the node equivalence table to find the corresponding unoptimized equivalent node. Then, compare the internal computational processes of the two nodes. If the computational process changes, the internal computational process of the equivalent node is determined to be the pattern that caused the compilation error. Otherwise, the node has no error pattern and only the error is scaled. 2) Use a fast binary search method to locate the erroneous optimization transformation: disable half of the suspicious transformations each time, that is, all transformations that perform optimization functions, recompile the erroneous pattern, and compare the recompiled running results with the running results of the pattern when the transformations are not disabled. This method is used to determine whether the transformation that caused the error is among the disabled transformations or the unselected transformations. Repeat the above binary search method until the erroneous optimization transformation is identified.

9. The automated debugging system for numerical errors of deep learning compilers according to claim 1, characterized in that: The verification means: disabling the erroneous optimization transformation obtained by positioning, completing other optimization transformations that are not disabled, and comparing the error of the new model with that of the unoptimized model. When the new error is reduced or disappears compared to the original, the verification module determines that the positioning is valid and outputs the positioning results and other debugging information, namely the error accumulation graph.

10. An automated debugging method based on the system according to any one of claims 1 to 9, characterized in that: After extracting the features of the defective model and constructing its calculation graph before and after optimization, each node in the graph is quantized into a hash value and node matching and operator calculation result comparison are performed to obtain the root cause of the error, including the generation and propagation of the error, the pattern that causes the error, and the compiler optimization transformation that causes the pattern rewriting.

Images