Precision error positioning method and device

Abstract

The invention provides a precision error positioning method and device, and belongs to the technical field of computers. In the embodiment of the invention, in response to determining that the precision of first output data of a first node in a tested network is abnormal, reference input data corresponding to the first node is input to the first node, and the first node is operated to obtain second output data of the first node; in response to determining that the precision of second output data of the first node is normal, intercepting a first sub-network from the tested network, inputting reference input data corresponding to the first sub-network to the first sub-network, and operating the first sub-network to obtain output data of the first sub-network, the first sub-network being an upper-layer network located at the first node in the tested network; in response to determining that the accuracy of the output data of the first sub-network is abnormal, it is determined that the first sub-network includes a cumulative error accuracy interval. According to the embodiment of the invention, accurate positioning of an accumulated precision error caused by an upper subnet is realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a precision error positioning method and device. Background Art

[0002] In a network, each layer of subnetwork is connected to the next layer of subnetwork, and the output data of the previous layer of subnetwork is the input data of the next layer of subnetwork. Therefore, if there is an accuracy error in the output data of the previous layer of subnetwork, the accuracy error of the previous layer of subnetwork will usually be transmitted to the next layer of subnetwork along with the network level, resulting in the continuous accumulation of accuracy errors of the entire network. This accumulated error will make the accuracy errors of subsequent subnetworks larger and larger.

[0003] Based on this, there is an urgent need to provide a solution that can locate the accumulated accuracy error caused by the upper subnet. Summary of the invention

[0004] The embodiment of the present application provides a method and device for locating an accuracy error, which can locate the accumulated accuracy error caused by the upper subnet. The technical solution is as follows.

[0005] In a first aspect, a method for locating an accuracy error is provided, the method comprising: in response to determining that the accuracy of first output data of a first node in a measured network is abnormal, inputting reference input data corresponding to the first node to the first node, and running the first node to obtain second output data of the first node; in response to determining that the accuracy of the second output data of the first node is normal, intercepting a first subnetwork from the measured network, inputting the reference input data corresponding to the first subnetwork to the first subnetwork, and running the first subnetwork to obtain output data of the first subnetwork, wherein the first subnetwork is an upper-layer network located at the first node in the measured network; in response to determining that the accuracy of the output data of the first subnetwork is abnormal, determining that the first subnetwork includes a cumulative error accuracy interval.

[0006] Since the reference input data can be understood as correct data, if the accuracy of the output data (second output data) generated by running the first node based on the correct data is abnormal, and the accuracy of the output data (first output data) generated by running the first node based on the output data of the previous node of the first node is normal, and the difference between the two runs of the first node is mainly the different input data used, it can be determined that the output data of the previous node causes the accuracy of the first node (second output data) to be abnormal, and the previous node of the first node or the node before the previous node (upper subnet) has a cumulative error accuracy interval. By intercepting the upper subnet of the first node, the data flow relationship between the first node and the upper network of the first node is disconnected, and the influence of the accuracy error of the output data of the upper network on the accuracy comparison of the first node is reduced, and then by inputting the corresponding reference input data to the upper subnet, if the accuracy of the output data of the upper network is abnormal, it is determined that the upper subnet includes a cumulative error accuracy interval, thereby realizing the precise positioning of the cumulative accuracy error caused by the upper subnet.

[0007] In a possible implementation, intercepting the first subnetwork from the network under test includes: determining a second subnetwork based on the network under test, the head node of the second subnetwork being the input node of the network under test, and the tail node of the second subnetwork being the first node; inputting reference input data corresponding to the second subnetwork into the second subnetwork, running the second subnetwork, and obtaining output data of the second subnetwork; and determining the first subnetwork from the second subnetwork based on a precision check result of the output data of the second subnetwork.

[0008] The first sub-network is further determined from the second sub-network based on the accuracy verification result of the second sub-network, thereby narrowing the range of the cumulative error accuracy interval from the second sub-network to the first sub-network, thereby improving the accuracy of the cumulative error accuracy interval.

[0009] In a possible implementation, the second subnetwork includes an intermediate node located between the head node and the tail node, and the reference input data corresponding to the second subnetwork is input into the second subnetwork, the second subnetwork is run, and the output data of the second subnetwork is obtained, including: inputting the reference input data corresponding to the intermediate node into the intermediate node, running the subnetwork from the intermediate node to the tail node in the second subnetwork, and obtaining the output data of the second subnetwork; and determining the first subnetwork from the second subnetwork based on the accuracy verification result of the output data of the second subnetwork includes: in response to determining that the accuracy of the output data of the second subnetwork is abnormal, determining the head node of the first subnetwork based on the intermediate node of the second subnetwork, and determining the tail node of the first subnetwork based on the tail node of the second subnetwork.

[0010] Since the verification results of the first sub-network from the middle node to the tail node have accuracy anomalies, it indicates that the middle node to the tail node includes the cumulative error accuracy interval. By resetting the head node, it is equivalent to moving the head node backward to the position of the middle node, and then re-dividing the first sub-network from the new middle node to the tail node into two, thereby reducing the range of the searched cumulative error accuracy interval by half, excluding the original range from the head node to the middle node, and improving the accuracy of the cumulative error accuracy interval.

[0011] In a possible implementation, in response to determining that the accuracy of the output data of the second subnetwork is normal, the tail node of the second subnetwork determines the tail node of the first subnetwork, and determines the head node of the first subnetwork based on the head node of the second subnetwork.

[0012] Since the accuracy of the verification result of the first sub-network is normal, it indicates that the cumulative error accuracy interval is not included from the middle node to the tail node. Therefore, the middle node to the tail node can be excluded from the range of the cumulative error accuracy interval. Since it has been verified before that the head node to the tail node includes the cumulative error accuracy interval, it can be determined that the cumulative error accuracy interval is located at the previous node from the head node to the middle node. By resetting the tail node, it is equivalent to moving the tail node forward to the previous node of the middle node, and then re-dividing the first sub-network from the head node to the new tail node into two, thereby reducing the range of the searched cumulative error accuracy interval by half, thereby improving the accuracy of the cumulative error accuracy interval.

[0013] In a possible implementation, determining the second subnetwork based on the network under test includes: determining a backbone node sequence from an input node of the network under test to the first node; and determining the second subnetwork from the backbone node sequence.

[0014] By determining the process of the trunk node sequence, the determined trunk node sequence includes a complete subnet, and the determined trunk node sequence is on the same straight line. The trunk node sequence structure is linear, without branches or redundant process nodes, thereby avoiding the need to execute additional input benchmark data for the process nodes and compare the output results of the process nodes due to the process nodes being cut into the subnet, thereby reducing the operational complexity of input data and comparing output results, and thereby improving the speed of positioning accuracy errors.

[0015] In a possible implementation manner, intercepting the first sub-network from the network under test includes: intercepting the parent node of the first node and the first node in the network under test to obtain the first sub-network.

[0016] By intercepting the parent node of the first node to the first node as the first sub-network, it helps to minimize the cumulative error accuracy range as much as possible, so as to locate the location of the cumulative error more quickly. Because the parent node is close to the node with the accuracy problem, it may be the node that directly affects the output result of the node with the accuracy problem, so starting to check from the parent node can effectively narrow the scope of the problem.

[0017] In one possible implementation, the method further includes: intercepting a third subnetwork from the network under test, inputting reference input data corresponding to the third subnetwork into the third subnetwork, running the third subnetwork, and obtaining output data of the third subnetwork, wherein the third subnetwork is a lower-layer network located at the first node in the network under test; in response to determining that the output data of the third subnetwork has an accuracy abnormality, determining that there is a node in the third subnetwork with an accuracy abnormality.

[0018] In a possible implementation, the method further includes: in response to determining that the precision of the second output data of the first node is abnormal, determining that the first node has a precision abnormality.

[0019] Since the output data generated by the first node based on the benchmark input data (correct data) still has an accuracy anomaly, it is determined that the first node itself has an accuracy anomaly. Based on this approach, it is possible to more accurately determine which specific operator the problem occurs in, without the need to debug the entire network, saving time and resources.

[0020] In a possible implementation, the first node is the Mth layer network in the network under test, the first subnetwork is the Kth layer network to the Mth layer network in the network under test, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0021] In one possible implementation, the network under test is a computing device cluster, which is used to execute a task sequence, the first node is a computing device or an AI acceleration card that executes the Mth task in the task sequence, and the first subnetwork is a computing device or an AI acceleration card that executes the Kth task to the Mth task in the task sequence, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0022] The above method is applicable to the scenario of locating the cumulative error precision interval in the computing device cluster. For example, in the scenario of distributed execution of a task sequence in the computing device cluster, based on the result of each computing device in the computing device cluster executing the corresponding task, one or more computing devices with precision problems when executing the task are determined from the computing device cluster.

[0023] In one possible implementation, the network under test includes a second node and a third node, the third node is a child node of the second node, and determining the backbone node sequence from the input node of the network under test to the first node includes: determining the flow value of the third node based on the flow value of the second node and the number of child nodes of the second node; in response to determining that the flow value of the third node is the same as the flow value of the input node, adding the third node to the backbone node sequence.

[0024] In one possible implementation, the first node includes a first instruction and a second instruction preceding the first instruction, and after determining that the first node has an accuracy abnormality, the method further includes: inputting reference input data corresponding to the first instruction into the first instruction, executing the first instruction, and obtaining output data of the first instruction; in response to determining that the accuracy of the output data of the first instruction is normal, inputting reference input data corresponding to the second instruction into the second instruction, executing the second instruction, and obtaining output data of the second instruction; in response to determining that the accuracy of the output data of the second instruction is abnormal, determining that the second instruction includes a cumulative error accuracy range.

[0025] In this way, a means of locating the precision of cumulative errors at the instruction level is provided, which helps to locate operators with precision problems and determine which instructions have accumulated the operator precision problems.

[0026] In a possible implementation, the first output data of the first node, the second output data of the first node, and the output data of the first subnetwork are all obtained through a first processor, and the benchmark input data corresponding to the first node, the benchmark input data corresponding to the first subnetwork, the benchmark output data corresponding to the first node, and the benchmark output data corresponding to the first subnetwork are all obtained through a second processor.

[0027] In one possible implementation, the determination that the accuracy of the first output data of the first node in the measured network is abnormal includes: determining that the difference between the first output data of the first node and the benchmark output data corresponding to the first node is greater than or equal to a threshold; the determination that the accuracy of the second output data of the first node is normal includes: determining that the difference between the second output data of the first node and the benchmark output data corresponding to the first node is less than or equal to the threshold; the determination that the accuracy of the output data of the first sub-network is abnormal includes: determining that the difference between the output data of the first sub-network and the benchmark output data corresponding to the first sub-network is greater than or equal to the threshold.

[0028] In a possible implementation, the first node is the Mth operator in the network under test, the first subnetwork is the Kth operator to the Mth operator in the network under test, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0029] In a possible implementation, the first node is the Mth instruction in the network under test, the first subnetwork is the Kth instruction to the Mth instruction in the network under test, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0030] In a possible implementation, the extracting the first subnetwork from the network under test includes: determining a topological structure of the first subnetwork based on a directed acyclic graph of the network under test; and determining the first subnetwork based on the topological structure of the first subnetwork.

[0031] Directed acyclic graph (DAG) can clearly reflect the computational process and dependencies in the network under test. By analyzing DAG, we can understand the data transfer and computation order between nodes, which helps to more accurately determine the nodes included in the first sub-network.

[0032] In one possible implementation, intercepting the first subnetwork from the network under test includes: in the process of running an inference service or a training service based on the network under test, executing the interception of the first subnetwork and the inference service or the training service in parallel. This method supports the simultaneous interception of the first subnetwork and the inference and training services, making full use of computing resources and speeding up the execution of the entire process. In addition, accuracy anomalies can be detected in real time during the inference or training process, which is more timely.

[0033] In a second aspect, a precision error positioning device is provided, the device comprising:

[0034] A determination module, configured to determine that the accuracy of first output data of a first node in a tested network is abnormal;

[0035] an operating module, configured to input the reference input data corresponding to the first node into the first node, and operate the first node to obtain second output data of the first node;

[0036] The determination module is further used to determine that the accuracy of the second output data of the first node is normal;

[0037] An interception module, used for intercepting a first sub-network from the network under test;

[0038] The operation module is further used to input the reference input data corresponding to the first sub-network into the first sub-network, and operate the first sub-network to obtain the output data of the first sub-network, wherein the first sub-network is an upper layer network located at the first node in the network under test;

[0039] The determination module is further configured to determine that the first sub-network includes a cumulative error accuracy interval in response to determining that the accuracy of the output data of the first sub-network is abnormal.

[0040] In a possible implementation, the interception module is used to determine a second subnetwork based on the network under test, the head node of the second subnetwork is the input node of the network under test, and the tail node of the second subnetwork is the first node; the reference input data corresponding to the second subnetwork is input into the second subnetwork, the second subnetwork is run, and the output data of the second subnetwork is obtained; based on the accuracy verification result of the output data of the second subnetwork, the first subnetwork is determined from the second subnetwork.

[0041] In a possible implementation, the second sub-network includes an intermediate node located between the head node and the tail node, and the running module is used to input the reference input data corresponding to the intermediate node into the intermediate node, run the sub-network from the intermediate node to the tail node in the second sub-network, and obtain the output data of the second sub-network;

[0042] The determination module is used to determine the head node of the first subnetwork based on the intermediate nodes of the second subnetwork and the tail node of the first subnetwork based on the tail node of the second subnetwork in response to determining that the accuracy of the output data of the second subnetwork is abnormal; or, in response to determining that the accuracy of the output data of the second subnetwork is normal, determine the tail node of the first subnetwork based on the intermediate nodes of the second subnetwork and the head node of the first subnetwork based on the head node of the second subnetwork.

[0043] In a possible implementation, the determination module is used to determine a backbone node sequence from an input node to the first node of the network under test; and determine a second sub-network from the backbone node sequence.

[0044] In a possible implementation manner, the determining module is configured to intercept the parent node of the first node and the first node in the tested network to obtain the first sub-network.

[0045] In a possible implementation, the interception module is further used to intercept a third sub-network from the network under test, input reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network, and obtain output data of the third sub-network, wherein the third sub-network is a lower layer network located at the first node in the network under test;

[0046] The determination module is further configured to determine that there are nodes with abnormal accuracy in the third sub-network in response to determining that the accuracy of the output data of the third sub-network is abnormal.

[0047] In a possible implementation, the determination module is further configured to determine that the first node has an accuracy abnormality in response to determining that the accuracy of the second output data of the first node is abnormal.

[0048] In one possible implementation, the network under test includes a second node and a third node, the third node is a child node of the second node, and the determination module is used to determine the flow value of the third node based on the flow value of the second node and the number of child nodes of the second node; in response to determining that the flow value of the third node is the same as the flow value of the input node, the third node is added to the trunk node sequence.

[0049] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the precision error positioning method as described in the first aspect or any implementation of the first aspect.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the precision error positioning method described in the first aspect or any implementation of the first aspect.

[0051] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the precision error positioning method described in the first aspect or any one of the implementations of the first aspect.

[0052] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of a precision error positioning method provided in an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of precision error positioning provided by an embodiment of the present application;

[0055] Figure 3 is a flow chart of a precision error positioning method provided in an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of determining a backbone node sequence provided by an embodiment of the present application;

[0057] Figure 5 It is a schematic diagram of searching for a cumulative error precision interval by using a binary search method provided in an embodiment of the present application;

[0058] Figure 6 It is a schematic diagram of determining the cumulative error accuracy interval using the three-part method provided in an embodiment of the present application;

[0059] Figure 7 is a flow chart of a precision error positioning method provided in an embodiment of the present application;

[0060] Figure 8 is a flow chart of a precision error positioning method provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of the structure of a precision error positioning device provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0064] Fig.12 A schematic diagram of the structure of another computer cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0066] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0067] (1) Accuracy cumulative error

[0068] Accumulated precision error refers to the precision problem caused by the accumulation of precision errors of multiple nodes (such as multi-layer operators), which causes the precision error of a node downstream of the multiple nodes (such as a layer of operators subsequent to the multi-layer operators) to exceed the set threshold.

[0069] (2) Artificial Intelligence (AI) Accelerator Card

[0070] AI accelerator cards are hardware used to perform AI computing tasks. The main function of AI accelerator cards is to provide efficient computing power and storage capacity to support large-scale deep learning model training and reasoning. For example, the types of AI accelerator cards include tensor processing units (TPUs), graphics processing units (GPUs), neural processing units (NPUs), Ascend AI processors or Cambrian AI processors, etc.

[0071] Attached Figure 1 Attached is a flow chart of a precision error positioning method provided in an embodiment of the present application. Figure 1 The method shown can be performed by a computing device cluster. For example, a computing device in the computing device cluster executes the attached Figure 1 In the method shown, each computing device in the computing device cluster is a node in the network under test; or, different computing devices in the computing device cluster interact to collaboratively execute the attached Figure 1 The method shown.

[0072] Step S100: The computing device obtains output data of each node of the tested network and benchmark data corresponding to each node.

[0073] The network under test includes a plurality of nodes. In a possible implementation, the network topology of the network under test has a plurality of layers. For example, the network under test includes an input layer, an intermediate layer, and an output layer. Each layer in the network under test has one or more nodes. The nodes of each layer are connected to the nodes of the next layer. The output data of the nodes of each layer are the input data of the nodes of the next layer. In an exemplary embodiment, the network topology of the network under test is a directed acyclic graph.

[0074] In an exemplary embodiment, the network under test is a machine learning model. For example, the network under test is a deep learning model. For example, the network under test is a convolutional neural network. Optionally, a node in the network under test is an operator. The granularity of the operator can be large or small. A large-granularity operator is, for example, an independent network. For example, the network under test includes a long short-term memory network (Long Short-Term Memory, LSTM) operator. The LSTM operator itself is also a network. A small-granularity operator is, for example, an operation instruction or a combination of multiple operation instructions. For example, a node (such as an operator) in the network under test includes at least one of a matrix operation instruction, a vector operation instruction, or a scalar operation instruction. For example, a node (such as an operator) in the network under test is a convolution operator, a fully connected operator, an activation function operator, a pooling operator, a normalization operator, a loss function operator, a deconvolution operator, or a depth-separable convolution operator. For another example, each node in the network under test is an interface, and the interface is obtained by encapsulating multiple operation instructions. For example, the network under test is a model trained based on frameworks such as PyTorch, ONNX, Caffe, TensorFlow, PaddlePaddle, or MindSpore.

[0075] In one possible implementation, the output data of each node of the network under test is a tensor. Tensors are used to represent multidimensional data. For example, a tensor is a scalar (0-dimensional), a vector (1-dimensional), a matrix (2-dimensional), or an array of higher dimensions. A tensor can also be called raw data. In the field of AI, the input data and output data of a network are usually represented in the form of tensors.

[0076] Benchmark data is also called golden data or benchmark data. Benchmark data acts as a benchmark and is used as a reference object for accuracy comparison. Benchmark data includes benchmark input data and benchmark output data. Benchmark output data is used to compare with the output data of the network under test. For example, the benchmark output data corresponding to the output layer node indicates the type of image. The benchmark output data corresponding to the middle layer node indicates the characteristics of the image.

[0077] In one possible implementation, the output data of each node in the network under test is generated by a first AI accelerator card. The benchmark data is generated by a second AI accelerator card. For example, a computing device includes a first AI accelerator card and a second AI accelerator card. The computing device runs the network under test through the first AI accelerator card to obtain the output data of each node in the network under test. The computing device runs the benchmark network through the second AI accelerator card to obtain the output data of each node in the benchmark network. For another example, a computing device cluster includes a first computing device and a second computing device, the first computing device includes a first AI accelerator card, and the second computing device includes a second AI accelerator card. The first computing device runs the network under test through the first AI accelerator card to obtain the output data of each node in the network under test. The second computing device runs the benchmark network through the second AI accelerator card to obtain the output data of each node in the benchmark network.

[0078] In one possible implementation, the reference network and the network under test are the same. For example, the reference network and the network under test have the same network structure, such as the reference network and the network under test have the same number of layers; for example, the nodes at the same layer in the reference network and the network under test have the same type; for example, the nodes at the same layer in the reference network and the network under test have the same number of sub-nodes. Since the reference network and the network under test have the same network structure, the errors caused by the differences in network structure are excluded during the subsequent accuracy comparison, which is more reliable.

[0079] In another possible implementation, the reference network is model-converted to obtain the network under test. For example, the data format of the model parameters (such as weight parameters, bias items, and other model parameters) in the reference network is converted into a data format suitable for the first AI accelerator card, and data type matching, shape conversion, and other operations are performed to obtain the network under test.

[0080] In one possible implementation, both the network under test and the reference network are image recognition models. The computing device inputs the same image into the network under test and the reference network respectively, recognizes the image through the network under test and the reference network respectively, obtains the output data of each node from the network under test, and obtains the reference data corresponding to each node from the reference network. Exemplarily, the output data of the operator obtained by running the model based on the GPU or the central processing unit (CPU) is used as the reference output data, and the output data of the operator generated by running the model by the Ascend AI processor is compared with the reference output data.

[0081] In one possible implementation, when the computing device is running the benchmark network, whenever it obtains output data of a node in the benchmark network, it dumps (saves) the output data of the node, for example, transferring, exporting or saving the output data of the node into a file, so that the output data of the node can be obtained later by accessing the saved file.

[0082] In step S110, the computing device compares the accuracy of the output data of each node in the tested network with the benchmark data corresponding to each node, so as to determine the nodes with abnormal accuracy in the tested network.

[0083] Taking the accuracy comparison of the first node in the network under test as an example, the computing device obtains the difference between the first output data of the first node and the reference output data corresponding to the first node to obtain the first difference. The computing device determines whether the first difference is greater than the set threshold. If the first difference is greater than the set threshold, it is determined that there is an accuracy abnormality in the first node (or that there is an accuracy problem in the first node), and step S120 is executed; if the first difference is less than or equal to the set threshold, it is determined that the accuracy of the first node is normal. Among them, the first node is any node in the network under test. The accuracy comparison process for other nodes other than the first node can refer to the accuracy comparison process for the first node.

[0084] In a possible implementation, the accuracy comparison process is a tensor comparison process. For example, the output data of the first node is a tensor, and the reference output data corresponding to the first node is also a tensor. The computing device compares the two tensors, determines the difference between the two tensors, and determines whether the difference is greater than a set threshold.

[0085] The way to determine the difference between two tensors includes at least one of absolute error, relative error, K2 divergence, cosine similarity, absolute relative error, maximum relative error, or mean relative error. Cosine similarity is a measurement method that measures the difference in direction and size between two vectors. When two tensors are regarded as vectors, cosine similarity can be used to determine the difference between the two tensors. Absolute error is a way to represent the difference between the elements of two tensors. Absolute error measures the difference between the two tensors by calculating the absolute value of the difference between the corresponding elements in the two tensors. Relative error refers to determining the difference between the two tensors by using the ratio of the absolute value of the difference between the elements of the two tensors to the mean of the corresponding elements in the two tensors. K2 divergence regards both tensors as probability distributions and uses K2 divergence to calculate the difference between the two tensors. Absolute relative error is an improved measure of relative error. Absolute relative error uses the ratio of the absolute value of the difference between two tensor elements to the absolute value of the corresponding elements in the two tensors to determine the difference between the two tensors. Maximum relative error is the maximum value of the relative error of all elements in the two tensors. Average relative error is the average value of the relative error of all elements in the two tensors.

[0086] In a possible implementation, the computing device traverses the output data of each node in the network under test; for the i-th node in the network under test, the computing device compares the accuracy of the output data of the i-th node in the network under test with the reference output data of the i-th node in the reference network; if the accuracy comparison result of the i-th node indicates an abnormality, the cumulative error accuracy interval is located for the upper subnet of the i-th node; if the accuracy comparison result of the i-th node indicates normal, the next node of the i-th node is traversed. i is a positive integer.

[0087] In one possible implementation, the computing device uses the output data generated by the second AI accelerator card running the network under test as the benchmark output data, and compares the output data generated by the first AI accelerator card running with the benchmark output data, so as to determine whether there is any accuracy problem in the output data generated by the second AI accelerator card running the network under test.

[0088] Step S120, in response to determining that the accuracy of the first output data of the first node in the tested network is abnormal, the computing device inputs the reference input data corresponding to the first node into the first node, runs the first node, and obtains the second output data of the first node.

[0089] This embodiment involves a process in which the same node is run multiple times. The data input to the node each time the node is run may be different, resulting in different output data obtained each time the node is run. In order to distinguish and describe the output data obtained by running the node at different times, "first output data" is used to describe the output data obtained by running the first node based on the output data of the previous node of the first node, and "second output data" is used to describe the output data obtained by running the first node based on the reference input data.

[0090] In a possible implementation, the computing device determines that the accuracy of the first output data of the first node is abnormal, then the computing device intercepts the first node from the network under test, thereby disconnecting the data flow relationship between the first node and the upper network of the first node, and reducing the influence of the accuracy error of the output data of the upper network on the accuracy comparison of the first node; the computing device obtains the reference input data corresponding to the first node from the reference network data, runs the first node to process the reference input data, and obtains the second output data output by the first node. Since the reference input data can be understood as correct data, if the accuracy of the output data (second output data) generated by running the first node based on the correct data is abnormal, and the accuracy of the output data (first output data) generated by running the first node based on the output data of the previous node of the first node is normal, and the difference between the two runs of the first node is mainly the different input data used, it can be determined that the output data of the previous node causes the accuracy of the first node (second output data) to be abnormal, and the previous node of the first node or the node before the previous node (upper subnet) has a cumulative error accuracy interval.

[0091] Step S140, in response to determining that the accuracy of the second output data of the first node is normal, the computing device intercepts the first subnetwork from the network under test, inputs the reference input data corresponding to the first subnetwork into the first subnetwork, and runs the first subnetwork to obtain the output data of the first subnetwork.

[0092] For example, the computing device obtains the difference between the second output data of the first node and the reference output data corresponding to the first node to obtain the second difference. The computing device compares the second difference with a set threshold. If the second difference is less than or equal to the set threshold, the computing device determines that the accuracy of the second output data is normal. If the second difference is greater than the set threshold, the computing device determines that the accuracy of the second output data is abnormal.

[0093] The first subnetwork is the upper network located at the first node in the network under test. For example, the first node is the Mth node, and the first subnetwork is the network composed of the Kth node to the Mth node in the network under test. For another example, the first node is the Mth layer network, and the first subnetwork is the network composed of the Kth layer network to the Mth layer network in the network under test. For another example, the first node is the Mth instruction, and the first subnetwork is the network composed of the Kth instruction to the Mth instruction in the network under test. M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0094] Since the accuracy problem occurs when the first node is run based on the output data of the previous node, and the accuracy problem does not occur when the first node is run based on the correct input data (reference input data), the possibility that the first node itself has an accuracy problem is ruled out, and it can be determined that the root cause of the accuracy problem lies in the upper subnetwork (first subnetwork) of the first node. In other words, the type of accuracy anomaly of the output data (first output data) generated by the first operation of the first node lies in the cumulative error. The accuracy analysis is performed on the first subnetwork by intercepting the first subnetwork to more accurately locate the accuracy problem.

[0095] In one possible implementation, a computing device obtains a directed acyclic graph (DAG) of the network under test; the computing device determines a network topology of the network under test based on the DAG of the network under test; the computing device determines an upstream and downstream relationship between nodes based on the network topology of the network under test; the computing device determines an upstream node of a first node based on the upstream and downstream relationship between the nodes, thereby determining a first subnetwork.

[0096] Step S160, in response to determining that the accuracy of the output data of the first sub-network is abnormal, determining that the first sub-network is within a cumulative error accuracy interval.

[0097] The computing device obtains the benchmark output data corresponding to the first sub-network from the benchmark network data; the benchmark output data corresponding to the first sub-network is, for example, the benchmark output data corresponding to the first node. For example, the first sub-network is the Kth node to the Mth node, and the benchmark output data corresponding to the first sub-network is, for example, the benchmark output data corresponding to the Mth node. The computing device obtains the difference between the output data of the first sub-network and the benchmark output data corresponding to the first sub-network, and the computing device determines whether the difference is greater than a set threshold. If the difference is greater than the set threshold, the computing device determines that the accuracy of the output data of the first sub-network is abnormal.

[0098] For example, please refer to the attached Figure 2 , the network under test is, for example, Figure 2 In the network 300, the first node is, for example, an attached Figure 2 The Mth layer node, i.e. Figure 2The first sub-network is, for example, an adjacent node 301. Figure 3 The middle network 310 is the network from the Kth layer node 302 to the node 301. In other words, the error precision interval is from the node 302 to the node 301. The node 302 is the starting point of the cumulative error, and the node 301 is the first node to detect the precision error.

[0099] Optionally, when determining that the first sub-network is within the cumulative error precision interval, the computing device replaces or repairs nodes with precision errors in the first sub-network, thereby improving the precision of the first sub-network. Alternatively, the computing device fuses some operators in the first sub-network to improve precision.

[0100] The method provided in this embodiment, when it is determined that the output data accuracy of a node is abnormal, replaces the input data of the node from the output data of the previous node with the benchmark input data. If the accuracy of the newly obtained output data is abnormal, it can be determined that the previous accuracy abnormality of the output data of the node was caused by the cumulative error. By intercepting the upper subnetwork of the node and inputting the corresponding benchmark input data into the upper subnetwork, if the accuracy of the output data of the upper network is abnormal, it is determined that the upper subnetwork includes a cumulative error accuracy interval, thereby achieving accurate positioning of the cumulative accuracy error.

[0101] Optionally, attach Figure 1 The method shown is applied to the scenario of locating the cumulative error precision interval in the deep learning model. Figure 1 The method shown is applied to the scenario of locating at least one of an operator with precision error, an instruction with precision error, or an interface with precision error in a deep learning model. Figure 1 The method shown is aimed at the scenario of locating the accuracy error in the process of performing reasoning or training business using deep learning models. Figure 1 The method shown is applied to the accuracy comparison of the output data obtained by running the same deep learning model through different hardware (such as different AI accelerator cards). Exemplarily, the first node is the Mth operator in the tested network, and the first subnetwork is the Kth operator to the Mth operator in the tested network, M is a positive integer greater than 1, and K is a positive integer less than or equal to M. For another example, the first node is the Mth layer network in the tested network, and the first subnetwork is the Kth layer network to the Mth layer network in the tested network, M is a positive integer greater than 1, and K is a positive integer less than or equal to M. In this way, it is possible to locate which operator in the entire network has an accuracy problem, or which operators have an accuracy problem caused by cumulative errors.

[0102] Optionally, attach Figure 1The method is applied to the scenario of locating the cumulative error precision interval in a computing device cluster. For example, in the scenario of distributed execution of a task sequence in a computing device cluster, based on the result of each computing device in the computing device cluster executing the corresponding task, one or more computing devices with precision problems are determined from the computing device cluster.

[0103] In another exemplary scenario, the network under test is a computing device cluster, which includes multiple computing devices, such as servers or terminals. Each computing device includes one or more AI accelerator cards. A node in the network under test is, for example, a computing device or an AI accelerator card in the computing device cluster. The computing device cluster is used to execute a task sequence. The task sequence includes multiple tasks, each of which is executed by a computing device or an AI accelerator card. For example, the task sequence is a model training task, and each task in the task sequence is a parameter update of the model. The first node is a computing device or an AI accelerator card that executes the Mth task in the task sequence, and the first subnetwork is a computing device or an AI accelerator card that executes the Kth to Mth tasks in the task sequence, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0104] Optionally, attach Figure 1 The method shown is applied to the scenario of locating the instruction interval with the cumulative error precision interval in the operator. For example, the first node is the Mth instruction in the tested network, the first subnetwork is the Kth instruction to the Mth instruction in the tested network, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0105] In a possible implementation, after determining that the output data accuracy of the first node is abnormal, the computing device further performs accuracy error positioning for the subsequent cascade subnet of the first node. Exemplarily, the process of performing accuracy positioning for the subsequent cascade subnet includes the following steps 1 to 3.

[0106] Step 1: The computing device intercepts the third sub-network from the network under test.

[0107] The third subnetwork is a lower-layer network located at the first node in the network under test. For example, the computing device determines each downstream node of the first node in the network under test based on the DAG of the network under test, and uses each downstream node of the first node as the third subnetwork. For example, the network under test includes N nodes, the first node is the Mth node in the network under test, and the third subnetwork is each downstream node of the Mth node in the network under test. For example, the head node of the third subnetwork is the (M+1)th node in the network under test, and the tail node of the third subnetwork is the Nth node in the network under test. For example, please refer to the attached Figure 2 , the third sub-network is, for example, Figure 2 Neutron Networks 320.

[0108] Since the third subnetwork is intercepted, the data flow relationship between the third subnetwork and the first node and the upper subnetwork of the first node is disconnected, making the third subnetwork a relatively independent subnetwork. Therefore, when the precision error analysis of the third subnetwork is performed subsequently, the output data of the nodes with precision problems in the first node and the upper subnetwork of the first node are reduced to interfere with the precision error positioning of the third subnetwork, thereby improving the accuracy of precision error positioning of the third subnetwork.

[0109] Step 2: The computing device inputs the reference input data corresponding to the third sub-network into the third sub-network; the computing device runs the third sub-network to obtain the output data of the third sub-network.

[0110] For example, the head node of the third sub-network is the (M+1)th node in the network under test, and the computing device inputs the reference input data corresponding to the (M+1)th node into the head node of the third sub-network (the (M+1)th node in the network under test). Since the input data of the third sub-network is replaced by the reference input data from the output data of the first node, the reference input data, which is a more accurate input data, can be used to detect whether the third sub-network itself has an accuracy problem.

[0111] Step three, in response to determining that the accuracy of the output data of the third sub-network is abnormal, determining that there are nodes in the third sub-network with abnormal accuracy.

[0112] For example, the difference between the output data of the third sub-network and the reference output data corresponding to the third sub-network (such as the output data corresponding to the last node in the network under test) is obtained to obtain a third difference. The computing device determines whether the third difference is greater than a set threshold. If the third difference is greater than the set threshold, the computing device determines that the accuracy of the output data of the third sub-network is abnormal; in addition, if the third difference is less than or equal to the set threshold, the computing device determines that the accuracy of the output data of the third sub-network is normal.

[0113] By providing the above-mentioned accuracy anomaly positioning method for the third subnetwork, considering that the output data of the nodes with accuracy problems in the measured network will flow into the subsequent cascade subnetworks, resulting in an impact on the accuracy of the nodes in the subsequent cascade subnetworks, resulting in the accuracy error of the subsequent cascade subnetworks of the nodes with accuracy problems cannot be accurately positioned, and by intercepting the subsequent cascade subnetworks of the nodes with accuracy problems, the data flow relationship between the subsequent cascade subnetworks and the nodes with accuracy problems is disconnected, and the input data of the subsequent cascade subnetworks is replaced with relatively correct input data (benchmark input data). When the input data are all benchmark input data, if the difference between the output data of the third subnetwork and the benchmark output data is greater than the threshold, it indicates that the third subnetwork itself has an accuracy problem, thereby accurately determining the nodes with accuracy anomalies in the third subnetwork.

[0114] Attached Figure 1 The method shown focuses on describing the situation where the accuracy of the output data (second output data) generated after re-running the first node based on the benchmark input data is normal. In other embodiments, if the accuracy of the output data (second output data) generated after re-running the first node based on the benchmark input data is abnormal, the computing device can determine that the first node itself has an accuracy abnormality. In other words, the error type of the first node is a single node error. It can also be understood that the cumulative error interval starts from the first node and ends at the first node.

[0115] Optionally, when the computing device determines that the first node has an accuracy anomaly, the computing device further locates the instruction with the accuracy anomaly in the first node; for example, the first node includes a first instruction and a second instruction before the first instruction, and the computing device inputs the reference input data corresponding to the first instruction into the first instruction, executes the first instruction, and obtains the output data of the first instruction; in response to determining that the accuracy of the output data of the first instruction is normal, the computing device inputs the reference input data corresponding to the second instruction into the second instruction, executes the second instruction, and obtains the output data of the second instruction; in response to determining that the accuracy of the output data of the second instruction is abnormal, it is determined that the second instruction includes a cumulative error accuracy interval. In this way, a means of locating the cumulative error accuracy at the instruction level is provided, which helps to locate which instructions accumulate the operator accuracy problem that causes the operator accuracy problem.

[0116] Attached Figure 3 Attached is a flow chart of a precision error positioning method provided in an embodiment of the present application. Figure 3 The method shown is to Figure 1 The overall flow chart of the cumulative error accuracy interval positioning process is shown in Figure 1, which adds the node error itself. Figure 3 The method shown supports nodes that output all accuracy errors in the network under test. Figure 3 In the method shown, the Mth layer node is attached Figure 1 A specific example of the first node in the method shown is attached Figure 3 The Kth to Mth subnetworks in the method shown are attached Figure 3 A specific example of the first sub-network in the illustrated method.

[0117] Step S210: The computing device obtains input data, runs the network under test based on the input data to generate network under test data, and runs the reference network based on the input data to generate reference network data.

[0118] The tested network data includes the output data of each layer of nodes in the tested network. The reference network data includes the output data of each layer of nodes in the reference network. In one possible implementation, the tested network and the reference network are the same. For example, the tested network and the reference network are the same neural network model, and the neural network model is run separately through different AI accelerator cards to obtain the tested network data and the reference network data.

[0119] In a possible implementation, the computing device includes a first processor and a second processor. Accordingly, step S210 includes step S211 and step S212.

[0120] Step S211: The computing device runs the network under test based on input data through the first processor, and saves the network under test data generated by each layer of nodes in the network under test.

[0121] Step S212: The computing device runs the benchmark network based on the input data through the second processor, and saves the benchmark network data generated by each layer of nodes in the benchmark network.

[0122] The input data used in step S211 and step S212 are, for example, the same data. For example, the network under test and the reference network are both image recognition models, and the input data used in step S211 and step S212 are the same image. For another example, the input data used in step S211 and step S212 are the same tensor. Since the input data used is the same, the accuracy error of the operator itself in the network can be measured by the difference in output data.

[0123] For example, the first processor is an NPU or an Ascend AI processor. The second processor is a GPU or a CPU. The benchmark network (the original network) is a framework network such as caffe, tensorflow, or onnx. The benchmark network is converted into an offline model supported by the first processor through tensor compilation, and the offline model is a specific example of the network under test. In addition, a fusion rule mapping file is output, and the fusion rule mapping file records the mapping relationship between the third-party framework and the nodes of om. The first processor loads the offline model (the network under test) and input data through ACL, performs reasoning based on the offline model and input data, and dumps the output data of each layer. The second processor runs the benchmark network, thereby dumping the output data of each layer. Since the default data type of the GPU or CPU is fp32 (a floating point type occupying 32 bits), and the data type on the Ascend AI processor is fp16 (a floating point type occupying 16 bits), the difference in the representation of the floating point data bit width causes different processors to have different precision for the output data of each layer of the network. Optionally, the output data of each layer of nodes dumped by the GPU or CPU is stored in the numpy format (npy).

[0124] Step S211 and step S212 may be performed sequentially. For example, step S211 may be performed first, and then step S212; or step S212 may be performed first, and then step S211. In other embodiments, step S211 and step S212 may also be performed in parallel, that is, step S211 and step S212 may be performed simultaneously.

[0125] Step S220: The computing device compares the accuracy of the output data of each layer of nodes in the tested network data with the output data of each layer of nodes in the reference network data, and determines that the output data of the Mth layer node in the tested network has an accuracy anomaly.

[0126] In a possible implementation, the output data of each layer of nodes in the tested network data are tensors (tensors are also called raw data), and the output data of each layer of nodes in the reference network data are tensors. Compare the output tensor of the i-th layer node in the tested network data with the output tensor of the i-th layer node in the reference network data. Obtain the difference between the output tensor of the i-th layer node in the tested network data and the output tensor of the i-th layer node in the reference network data. If the difference between the two output tensors is greater than or equal to the difference threshold, it is determined that the output data accuracy of the i-th layer node is abnormal. If the difference between the two output tensors is less than or equal to the difference threshold, it is determined that the output data accuracy of the i-th layer node is normal. i is a positive integer greater than or equal to 1 and less than or equal to M.

[0127] The difference between the two output tensors may be determined by, for example, absolute error, relative error, K2 divergence, cosine similarity, absolute relative error or average relative error, etc. The difference threshold may be determined based on experience or a business scenario applicable to the tested network.

[0128] In a possible implementation, the computing device starts from the first layer of nodes in the network data under test, traverses each layer of nodes in the network data under test, and for the traversed i-th layer of nodes in the network data under test, performs an accuracy comparison between the output data of the i-th layer of nodes in the network data under test and the output data of the i-th layer of nodes in the reference network data. If the output data of the i-th layer of nodes in the network under test has an accuracy anomaly, step S240 is performed for the i-th layer in the network under test, where i is a specific example of M, and the M layer is equivalent to a first-level accuracy anomaly point. If the output data accuracy of the i-th layer of nodes in the network under test is normal, the output data of the (i+1)th layer of nodes in the network under test continues to be traversed until the output data of the last layer of nodes in the network under test is traversed.

[0129] Step S240: For the M-th layer node in the network data under test, the computing device obtains the benchmark input data corresponding to the M-th layer node from the benchmark network data, inputs the benchmark input data corresponding to the M-th layer node into the M-th layer node, and re-runs the M-th layer node through the first processor to process the benchmark input data corresponding to the M-th layer node, and obtains the regenerated output data (second output data) of the M-th layer node. The computing device obtains the benchmark output data of the M-th layer node from the benchmark network data; the computing device performs an accuracy comparison between the regenerated output data of the M-th layer node and the benchmark output data of the M-th layer.

[0130] The Mth layer node is, for example, one node. In other words, the Mth layer node is a single-node network. The reference input data corresponding to the Mth layer node is, for example, output data generated by the (M-1)th layer node in the reference network after the second processor runs the reference network.

[0131] If the output data accuracy of the regenerated M-th layer node is normal, the computing device can determine that the reason for the accuracy abnormality of the output data of the M-th layer node is the cumulative error, that is, the output data accuracy abnormality of the M-th layer node generated for the first time is due to the existence of the cumulative error accuracy interval in the upper subnetwork of the M-th layer node, rather than the M-th layer node itself causing the output data accuracy abnormality of the M-th layer node generated for the first time. In this case, further execute step S260.

[0132] If the output data precision of the regenerated Mth layer node is abnormal, the computing device determines that the Mth layer node itself has a cumulative precision error. In other words, the computing device determines that the precision error type is a single node error, in which case the computing device further performs step S250.

[0133] Step S250: the computing device records the information of the Mth layer nodes.

[0134] Step S260: The computing device performs precision error positioning on subsequent cascaded subnets of the Mth layer nodes.

[0135] Step S260 includes the following steps S261 to S263.

[0136] Step S261: The computing device intercepts the subnetworks from the Kth layer to the Mth layer in the network under test.

[0137] In the process of selecting the K layer, the computing device first selects the parent node of the M layer in the network under test (the (M-1) layer node) as the K layer, and the computing device inputs the benchmark input data corresponding to the (M-1) layer to the (M-1) layer node. If the output data accuracy of the M layer node is abnormal, the computing device determines that the (M-1) layer node has an accuracy abnormality. If the output data accuracy of the M layer node is normal, the computing device determines that the accuracy abnormality of the M layer node is caused by the upper layer subnet of the (M-1) layer node.

[0138] In step S262, the computing device determines the backbone node sequence from the input node of the network under test to the Mth layer node using a linearization algorithm. Afterwards, the computing device searches for the cumulative error precision interval from the backbone node sequence using a search algorithm (e.g., a binary search algorithm). Optionally, when multiple sub-networks are searched out, the computing device searches for the cumulative error precision interval from the multiple sub-networks in parallel.

[0139] Step S263, the computing device obtains the benchmark input data corresponding to the K-th layer node from the benchmark network data. The computing device inputs the benchmark input data corresponding to the K-th layer to the intercepted K-th layer node to the M-th layer node. The computing device reruns the K-th layer to the M-th layer node through the first processor to process the benchmark input data corresponding to the K-th layer, and obtains the regenerated output data (second output data) of the M-th layer subnetwork. The computing device compares the output data of the regenerated M-th layer node with the benchmark output data corresponding to the M-th layer node. By selecting the K layer, the range of KM is gradually narrowed, and the accurate positioning of the cumulative error is achieved.

[0140] If the output data of the regenerated sub-network from the Kth layer to the Mth layer has accuracy anomalies, the cumulative error accuracy interval determines that the sub-network from the Kth layer to the Mth layer is a cumulative error interval.

[0141] Optionally, if the Pth layer located between the Kth layer and the Mth layer in the subnetwork from the Kth layer to the Mth layer also needs input data, the reference input data corresponding to the Pth layer is obtained from the reference network data, and the reference input data corresponding to the Pth layer is input into the Pth layer in the subnetwork from the Kth layer to the Mth layer.

[0142] In step S280, the computing device obtains the benchmark input data corresponding to the (M+1)th layer of subnetwork from the benchmark network data, inputs the benchmark input data corresponding to the (M+1)th layer of subnetwork into the (M+1)th layer of subnetwork in the network under test, and re-runs the (M+1)th layer of subnetwork to the last layer of subnetwork through the first processor to obtain the output data of the last layer of subnetwork, analyzes the accuracy problem of the subsequent subnetwork of the Mth layer based on the output data of the last layer of subnetwork and the benchmark output data corresponding to the last layer of subnetwork, and re-executes step S220.

[0143] In one possible implementation, in the attached Figure 1 In the method shown, the computing device determines the first sub-network by a linearization algorithm. Figure 3 In the method shown, the computing device uses a linearization algorithm to search for the cumulative error accuracy interval from the K-layer to the M-layer sub-network.

[0144] For example, in the attached Figure 1 In the method shown, the computing device determines a backbone node sequence from the input node to the first node of the network under test through a linearization algorithm; determines a second sub-network from the backbone node sequence; in other words, each node in the second sub-network is a backbone node in the network under test, and the first sub-network is a sub-node sequence in the backbone node sequence.

[0145] In one possible implementation, the network under test includes a second node and a third node, the third node is a child node of the second node, and the flow value of the third node is determined based on the flow value of the second node and the number of child nodes of the second node; in response to determining that the flow value of the third node is the same as the flow value of the input node, the third node is added to the backbone node sequence.

[0146] In one possible implementation, in order to determine the backbone node sequence, the computing device traverses each node from the input node to the Mth layer node in the network under test. For the currently traversed node, the computing device determines the flow value of the current node and compares the flow value of the current node with the flow value of the input node. If the flow value of the current node is equal to the flow value of the input node, the computing device adds the current node to the backbone node sequence. In addition, based on the flow value of the current node and the out-degree of the current node, the computing device allocates the flow value to each child node of the current node in proportion. The flow value allocated to each child node is, for example, the ratio between the flow value of the current node and the out-degree of the current node. Exemplarily, the flow value of a node is the sum of the ratios between the flow value of each parent node of the node and the out-degree of the corresponding parent node.

[0147] The flow value is used to describe the information transmitted between nodes in the network. The flow value represents the dependency and data flow between nodes. The in-degree of a node is the number of child nodes of the node. The in-degree of a node can also be understood as the number of edges pointing to the node. The out-degree of a node is the number of parent nodes of the node. The out-degree of a node can also be understood as the number of edges pointing from the node to other nodes. In-degree and out-degree are used to describe the connectivity and dependency of nodes in the network.

[0148] Exemplarily, the process of determining the backbone node sequence includes the following steps A to D.

[0149] Step A: according to a given constant C, set the flow value flow=C. The flow value flow is usually the in-degree value of the input node, which is used to determine the condition of the trunk node.

[0150] Step B, initialize the flow value nodeflow[node] of each node from the input node to the Mth node to 0.0, and initialize the flow value of the input node to C.

[0151] Step C, for each node from the input node to the Mth node, calculate the flow value nodeflow[node] of the node.

[0152] For example, the out-degree value of each node is out[node], and the in-degree value of each node is in[node]. The starting point is added to the BFS queue and a breadth-first search (BFS) is started, as shown in step 4.

[0153] Step D, establish a BFS queue, which is used to save the backbone node sequence. Traverse each node from the input node to the Mth node, and for the currently traversed node, determine whether the flow value nodeflow[node] of the current node is equal to the flow value flow. If the flow value nodeflow[node] of the current node is equal to the flow value flow, the current node is determined as the backbone node and the current node is added to the backbone node sequence. In addition, based on the flow value nodeflow[son] of the current node and the out-degree out[node] of the current node, determine the flow value of each child node of the current node.

[0154] For example, please refer to the attached Figure 4 , attached Figure 4 A specific example of using the linearization algorithm to determine the trunk node sequence is shown. Figure 4The network 400 shown is a specific example of the network under test, and the input node (starting node, first-layer node) of the network 400 is node 410. The flow value of node 410 is 3. Node 410 has three child nodes, and the three child nodes of node 410 are node 421, node 422 and node 350. The out-degree of node 410 is 3. The ratio between the flow value (3) of node 410 and the out-degree (3) of node 410 is 1. Therefore, the flow values ​​assigned by node 410 to nodes 421, node 422 and node 350 are all 1. The flow values ​​of nodes 421 and node 422 are both 1. Node 421 has two child nodes. The two child nodes of node 421 are node 431 and node 432. The out-degree of node 421 is 2. The ratio between the flow value (1) of node 421 and the out-degree (2) of node 421 is one-half, so the flow value that node 421 allocates to node 431 and node 432 is one-half. The flow values ​​of node 431 and node 432 are both one-half. The child node of node 431 and the child node of node 432 are both node 441. The out-degree of node 431 and the out-degree of node 432 are both 1, so the flow value that node 431 and node 432 allocate to node 441 is one-half. The flow value allocated to node 441 is the sum of the flow value of node 431 (one-half) and the flow value of node 432 (one-half), and the flow value of node 441 is 1. The child node of node 422 is node 442, and the ratio between the flow value (1) of node 422 and the out-degree (1) of node 422 is 1, so the flow value allocated to node 442 is 1. The flow value assigned by node 441 to node 450 is 1. The flow value assigned by node 442 to node 450 is 1. The flow value of node 450 is 1+1+1=3. The child node of node 450 is node 460, and the flow value assigned by node 450 to node 460 is 3. In network 400, the flow values ​​of nodes 410, 450, and 460 are all equal to the set initial flow value (3), so nodes 410, 450, and 460 are added to the backbone node sequence, and the resulting backbone node sequence includes nodes 410, 450, and 460.

[0155] By determining the process of the trunk node sequence, the determined trunk node sequence includes a complete subnet, and the determined trunk node sequence is on the same straight line. The trunk node sequence structure is linear, without branches or redundant process nodes, thereby avoiding the need to execute additional input benchmark data for the process nodes and compare the output results of the process nodes due to the process nodes being cut into the subnet, thereby reducing the operational complexity of input data and comparing output results, and thereby improving the speed of positioning accuracy errors.

[0156] In one possible implementation, in the attached Figure 1In the method shown, the computing device determines the first sub-network in a recursive manner. Figure 3 In the method shown, the computing device recursively searches for the cumulative error accuracy interval from the K-layer to the M-layer sub-network.

[0157] For example, the computing device determines the second sub-network based on the network under test, the head node of the second sub-network is the input node of the network under test, and the tail node of the second sub-network is the first node; the computing device inputs the reference input data corresponding to the second sub-network into the second sub-network, and the computing device runs the second sub-network to obtain the output data of the second sub-network; the computing device determines the first sub-network from the second sub-network based on the accuracy verification result of the output data of the second sub-network. By further determining the first sub-network from the second sub-network based on the accuracy verification result of the second sub-network, the range of the cumulative error accuracy interval is narrowed from the second sub-network to the first sub-network, thereby improving the accuracy of the cumulative error accuracy interval.

[0158] In a possible implementation, the computing device determines the first sub-network based on the accuracy check result of the interval from the intermediate node to the end node in the second sub-network. Specifically, the computing device determines the intermediate node of the second sub-network based on the head node of the second sub-network and the tail node of the second sub-network. The computing device intercepts the network from the intermediate node of the second sub-network to the first node of the second sub-network to obtain the fourth sub-network. The computing device inputs the reference input data corresponding to the intermediate node into the intermediate node, runs the fourth sub-network, and obtains the third output data of the first node; in response to determining that the accuracy of the third output data of the first node is abnormal, the head node of the first sub-network is determined based on the intermediate node of the second sub-network, and the tail node of the first sub-network is determined based on the tail node of the second sub-network; or, in response to determining that the accuracy of the third output data of the first node is normal, the tail node of the first sub-network is determined based on the intermediate node of the second sub-network, and the head node of the first sub-network is determined based on the head node of the second sub-network.

[0159] In a possible implementation, the computing device uses a binary algorithm to locate the first sub-network, which is described below by way of example through steps one and two.

[0160] Step 1: The computing device sets the head node of the second sub-network and the tail node of the second sub-network.

[0161] The computing device sets the input node of the network under test as the head node of the second sub-network. The computing device sets the node (eg, the Mth layer node) whose output data has an accuracy anomaly as the tail node of the second sub-network.

[0162] In step 2, the computing device executes a binary algorithm to iteratively update the position of the intermediate node in the second sub-network until the position of the head node is before the tail node, the position of the head node is the same as the tail node, or the position of the intermediate node is the same as the head node, thereby gradually narrowing the scope of the second sub-network.

[0163] For example, step 2 includes step (2-1) to step (2-4).

[0164] In step (2-1), the computing device calculates the position of the intermediate node of the second sub-network based on the position of the head node of the second sub-network and the position of the tail node of the second sub-network.

[0165] For example, the midpoint between the position of the head node of the second sub-network and the position of the tail node of the second sub-network is determined as the position of the intermediate node of the second sub-network. For example, the position of the intermediate node is obtained by the following formula (1).

[0166] mid=(head+tail+1) / 2;(1)

[0167] Among them, mid represents the position of the middle node, head represents the position of the head node, and tail represents the position of the tail node.

[0168] Step (2-2) The computing device intercepts the node from the intermediate node to the output data accuracy abnormality from the second sub-network to obtain the first sub-network. The computing device checks whether the accuracy of the first sub-network is normal. For example, the computing device inputs the benchmark input data corresponding to the intermediate node to the first sub-network, executes the first sub-network, and obtains the output data. The computing device compares the output data with the benchmark output data. If the difference between the output data and the benchmark output data is greater than or equal to the threshold, the computing device determines that the verification result is an accuracy abnormality, that is, there is an accuracy error problem. If the difference between the output data and the benchmark output data is less than or equal to the threshold, the computing device determines that the verification result is normal accuracy.

[0169] In step (2-3), if the verification result is that the accuracy is abnormal, the computing device sets the intermediate node of the second sub-network as the new head node. In other words, the head node of the first sub-network is the intermediate node of the second sub-network.

[0170] Since the verification results of the first sub-network from the middle node to the tail node have accuracy anomalies, it indicates that the middle node to the tail node includes the cumulative error accuracy interval. By resetting the head node, it is equivalent to moving the head node backward to the position of the middle node, and then re-dividing the first sub-network from the new middle node to the tail node into two, thereby reducing the range of the searched cumulative error accuracy interval by half, excluding the original range from the head node to the middle node, and improving the accuracy of the cumulative error accuracy interval.

[0171] In step (2-4), if the test result is that the accuracy is normal, the computing device sets the previous node of the middle node of the second sub-network as the new tail node. In other words, the tail node of the first sub-network is the previous node of the middle node of the second sub-network.

[0172] Since the accuracy of the verification result of the first sub-network is normal, it indicates that the cumulative error accuracy interval is not included from the middle node to the tail node. Therefore, the middle node to the tail node can be excluded from the range of the cumulative error accuracy interval. Since it has been verified before that the head node to the tail node includes the cumulative error accuracy interval, it can be determined that the cumulative error accuracy interval is located at the previous node from the head node to the middle node. By resetting the tail node, it is equivalent to moving the tail node forward to the previous node of the middle node, and then re-dividing the first sub-network from the head node to the new tail node into two, thereby reducing the range of the searched cumulative error accuracy interval by half, thereby improving the accuracy of the cumulative error accuracy interval.

[0173] For example, please refer to the attached Figure 5 , attached Figure 5 The following is a specific example of searching for the cumulative error precision interval using a binary search method. Figure 5 The following diagram shows three binary search processes. In the first binary search process, as shown in the figure Figure 5 As shown in (a) in the figure, during the process of executing step (2-2), the computing device intercepts the sub-network from the intermediate node 512 to the tail node 513 from the network under test to obtain the sub-network 510; the computing device performs an accuracy check on the sub-network 510; the computing device determines that the accuracy of the sub-network 510 is normal, then executes step (2-4), sets the previous node of the intermediate node 512 as the new tail node, and thus determines the tail node 523. In the second binary search process, as shown in the attached figure Figure 5 As shown in (b) in the figure, the computing device determines the intermediate node 522 based on the head node 511 and the tail node 523 during the execution of step (2-1). During the execution of step (2-2), the computing device intercepts the network from the intermediate node 522 to the terminal node from the network under test to obtain the sub-network 520. During the execution of step (2-2), the computing device performs an accuracy check on the sub-network 520. If the computing device determines that the accuracy of the sub-network 520 is normal, it executes step (2-4) to set the previous node of the intermediate node 522 as the new tail node, thereby determining the tail node 533. In the third binary search process, as shown in the attached figure, Figure 5As shown in (c), the computing device determines the intermediate node 532 based on the head node 511 and the tail node 533 during the execution of step (2-1). The computing device intercepts the network from the intermediate node 532 to the terminal node from the network under test during the execution of step (2-2) to obtain the sub-network 530. During the execution of step (2-2), the computing device performs an accuracy check on the sub-network 530. If the computing device determines that the accuracy of the sub-network 530 is abnormal, it executes step (2-3) to set the intermediate node 532 as the new head node, thereby determining the head node 541. The computing device outputs the identifier of the head node 541 and the identifier of the terminal node, indicating that the cumulative accuracy error interval is from the head node 541 to the terminal node.

[0174] Exemplarily, the computing device executes the following program code to determine the cumulative accuracy error interval.

[0175] while head <tail:

[0176] mid=(head+tail+1) / 2

[0177] #Check the connectivity graph from mid node to endnode node

[0178] #Is the accuracy normal?

[0179] res=check(mid,endnode)

[0180] # Accuracy exception

[0181] if res:

[0182] head=mid

[0183] # Accuracy is correct

[0184] else:

[0185] tail=mid–1

[0186] #Return the starting node and ending node of the precision error interval

[0187] return head,endnode

[0188] In another possible implementation, the computing device uses the three-part method to determine the cumulative error accuracy interval. Figure 6As shown, the computing device determines the quantile α and the quantile (1-α) from the current search interval (such as the second subnetwork) according to the proportional constant α. The proportional constant α is generally taken as one-third. The quantile α refers to the point obtained by dividing the current search interval according to the proportional constant α. The calculation method of the quantile α is, for example, to weight the starting point and the end point of the current search interval according to the proportion of α. In other words, the quantile α divides the entire search interval into the α part and the (1-α) part, and the α quantile is located at the position occupied by the α part of the entire search interval. The (1-α) quantile is located at the position occupied by the (1-α) part of the entire search interval.

[0189] The quantile α is determined by the following formula (2).

[0190] α-divide=(1-α)*head+α*tail; (2)

[0191] The quantile (1-α) is determined by the following formula (3).

[0192] (1-α)-divide=α*head+(1-α)*tail; (3)

[0193] Execute the check(startnode,endnode) function to perform a precision check on the interval [startnode,endnode]. If the check function returns true, it indicates that the precision of the interval [startnode,endnode] is abnormal; if the check function returns false, it indicates that the precision of the interval [startnode,endnode] is normal. In addition, the next search interval is determined based on the interval precision check results check(α,endnode) and check(1-α,endnode). Specifically, if check(α,endnode) returns true and check(1-α,endnode) returns true, that is, the interval (α,endnode) and the interval (1-α,endnode) are both abnormal in precision, then the next search interval is [(1-α)-divide,tail]. If check(α,endnode) returns true and check(1-α,endnode) returns false, that is, the interval (α,endnode) is abnormal in precision and the interval (1-α,endnode) is normal in precision, then the next search interval is [α-divide,(1-α)-divide). If check(α,endnode) returns false and check(1-α,endnode) returns false, that is, the accuracy of the interval (α,endnode) is normal and the accuracy of the interval (1-α,endnode) is normal, then the next search interval is [head,α-divide).

[0194] In another possible implementation, the parent node of the first node and the first node in the tested network are intercepted to obtain the first sub-network.

[0195] In an exemplary scenario, the network under test is an image recognition model. The computing device inputs the original image into the image recognition model, recognizes the original image through the image recognition model, and obtains the output data of each node in the image recognition model. For example, the image recognition model includes input layer nodes, convolution layer nodes, pooling layer nodes, and output layer nodes. The output data of the input layer nodes represent the pixels in the original image or the features of the original image. The convolution layer nodes perform convolution operations on the original image through convolution kernels and output local features in the original image. The pooling layer nodes output the most significant features in the feature map. The output data of the output layer nodes is a probability value, which represents the possibility that the image belongs to a type.

[0196] In an exemplary scenario, the network under test is a text recognition model. The computing device inputs the original text into the text recognition model, and recognizes the original text through the text recognition model to obtain the output data of each node. The input layer node converts the original text into a character, word or encoded vector representation. The output data of each input layer node is, for example, a character, word or feature in the original text. The embedding layer node maps the discretely represented characters, words or features to a continuous low-dimensional vector space, and outputs a word embedding vector or a character embedding vector. The convolution layer performs a convolution operation on the input sequence through a sliding convolution kernel to extract local combined features. Each node corresponds to a specific feature extracted by a convolution kernel at a certain position in the input sequence. The recurrent layer node extracts the temporal relationship of the input sequence through a long short-term memory (LSTM) or a gated recurrent unit (GRU), and models the previous and next context information based on the temporal relationship. The attention layer node is used to assign weights to different parts of the input sequence and combine the attention-weighted representation with the context information. The output data of each attention layer node is, for example, an attention-weighted representation or context vector.

[0197] Attached Figure 7 Attached is a flow chart of a precision error positioning method provided in an embodiment of the present application. Figure 7 The method shown is a system-level precision error location method. Figure 7 The method shown supports parallel monitoring of the accuracy of each layer of operators in the AI ​​model during the execution of training or reasoning services based on the AI ​​model. Figure 7 The method shown in the figure can analyze the accuracy of operators at each layer in the AI ​​model in real time when the computing device performs training or reasoning services based on the AI ​​model. Figure 7 The method shown can locate the precision error of operators and can also locate the precision error of sub-networks. Figure 7 It is attached Figure 3 The method shown is applied to a specific example of an AI training and reasoning business scenario. Figure 7 The network under test in the method shown is, for example, an AI model. Figure 7 The output data of each layer of the network in the method shown is attached Figure 3 Specific examples of output data of the subnetwork in the method shown are attached. Figure 7 The operators in the methods shown are specific examples of nodes. Figure 7 The method shown includes the following steps.

[0198] Step S601: The computing device performs DAG construction on the network under test to obtain the DAG of the network under test.

[0199] DAG indicates the topology of the network under test. DAG includes information about nodes at each layer in the network under test. DAG is performed so that a subgraph can be intercepted based on DAG.

[0200] Step S602: The computing device runs the network under test to obtain output data of each layer in the network under test. The computing device saves the output data of each layer in the network under test to the memory. The computing device can save the output data of each layer in the network under test to the hard disk, or save the output data of each layer in the network under test to the memory.

[0201] In one possible implementation, the computing device detects the life cycle of the output data of each layer stored in the memory; in response to detecting that the life cycle of the output data of each layer has ended, the computing device deletes the stored output data of each layer from the memory, thereby saving occupied storage space.

[0202] Step S603: The computing device performs a precision comparison between the output data of each layer of operators in the tested network data and the output data of each layer of operators in the benchmark data based on the stored tested network data and benchmark data.

[0203] Step S604: If the first operator with precision problem in the tested network data is located based on the precision comparison result, the computing device extracts the first operator with precision problem from the tested network data, inputs the reference input data to the operator, executes the operator, and obtains the output data of the operator. The computing device compares the output data of the operator with the reference output data corresponding to the operator. Based on the comparison result, the computing device determines whether the operator has precision problem.

[0204] Step S605: If the extracted single operator has no precision problem, the computing device determines that the precision error of the layer operator is caused by the cumulative error.

[0205] Step S606: The computing device divides the network under test into one or more subgraphs based on the DAG.

[0206] Step S607: The computing device extracts a backbone node sequence from one or more subgraphs of the network under test by using a linearization algorithm.

[0207] Step S608: The computing device inputs reference data into the trunk node sequence, executes the trunk node sequence, and obtains output data of the trunk node sequence.

[0208] Step S609: The computing device compares the output data of the trunk node sequence with the reference output data of the tail node corresponding to the trunk node sequence.

[0209] Step S610: The computing device locates the cumulative accuracy error interval based on the comparison result.

[0210] Optionally, when multiple trunk node sequences are extracted, the computing device executes the precision error positioning process of the multiple trunk node sequences in parallel. For example, the computing device inputs the reference input data to the head node of each trunk node sequence in parallel, and the computing device executes each trunk node sequence in parallel to obtain the output data of the tail node of each trunk node sequence; the computing device compares the output data of the tail node of each trunk node sequence with the reference output data corresponding to the tail node in parallel, thereby determining in parallel whether each trunk node sequence includes a cumulative precision error interval.

[0211] Optionally, the computing device uses a recursive method to further determine the child node sequence from the trunk node sequence, and judges whether the child node sequence includes a cumulative precision error interval based on the output data of the child node sequence, thereby narrowing the range of the determined cumulative precision error interval and improving the accuracy of the cumulative precision error interval.

[0212] Step S611: The computing device uses a search algorithm such as binary search to extract a sub-node sequence from a trunk node sequence.

[0213] Step S612: the computing device inputs the benchmark data into the sub-node sequence, executes the sub-node sequence, and obtains the output data of the sub-node sequence.

[0214] Step S613: the computing device compares the output data of the sub-node sequence with the reference output data of the sub-node sequence.

[0215] In step S614, the computing device determines whether the sub-node sequence includes a cumulative error precision interval based on the comparison result, thereby locating the cumulative error precision interval.

[0216] Step S615: If there is an accuracy problem with the output data of a single operator, the computing device extracts the benchmark input data of the operator from the benchmark network data, inputs the benchmark input data to the operator, and executes the operator to obtain the output data of the operator.

[0217] Step S616: The computing device compares the output data of the operator with the reference output data corresponding to the operator to obtain a comparison result.

[0218] Step S617: The computing device determines, based on the comparison result, that the accuracy problem of the output data of the operator is caused by the operator itself.

[0219] Step S618: The computing device inputs the benchmark data to the subsequent cascade subnetwork of the abnormal operator.

[0220] Step S619: The computing device runs subsequent cascaded subnetworks based on the benchmark data, thereby analyzing the accuracy of the subsequent cascaded subnetworks.

[0221] Step S621: If a single operator itself has precision problems, the computing device performs DAG construction on the matrix operation instructions, vector operation instructions, and scalar operation instructions contained in the operator.

[0222] Step S622: The computing device inputs a reference input tensor to the operator, executes the operator, and obtains an output tensor of the operator; the computing device compares the output tensor of the operator with the output tensor of the CPU.

[0223] In step S623, the computing device uses a search algorithm such as binary search to intercept some instructions from the DAG of the operator; the computing device executes the intercepted instructions through the first processor based on the benchmark input tensor to obtain the output tensor of the instructions; or, the computing device intercepts the upper-level interface of the instruction encapsulation from the DAG of the operator, the computing device inputs the benchmark input tensor into the interface, calls the interface through the first processor, and obtains the output tensor of the interface.

[0224] In step S624, the computing device compares the output tensor of the first processor executing the instruction with the output tensor of the second processor (e.g., CPU) executing the instruction to obtain a comparison result. Alternatively, the computing device compares the output tensor of the interface called by the first processor with the output tensor of the interface called by the second processor (e.g., CPU) to obtain a comparison result.

[0225] Step S625: If the comparison result of the output tensor of the instruction indicates that the precision is abnormal, the computing device determines that the intercepted instruction includes the cumulative error precision interval, thereby identifying which instructions cause the precision of the operator to have an error. If the comparison result of the output tensor of the interface indicates that the precision is abnormal, the computing device determines that the intercepted interface includes the cumulative error precision interval, thereby identifying which interfaces cause the precision of the operator to have an error.

[0226] Step S618: The computing device inputs reference input data corresponding to the subsequent cascade subnetwork to the subsequent cascade subnetwork of the operator whose output data has precision anomaly.

[0227] Step S619: The computing device runs the subsequent cascaded subnets to determine the accuracy of the subsequent cascaded subnets.

[0228] The method provided in this embodiment compares the actual output data of the operator, instruction, interface and network levels with the corresponding benchmark output data to achieve a comprehensive analysis of the precision error operators of the entire network. This method can directly locate which layer of the operator in the entire network has a precision problem, or which layers of operators have accumulated errors that cause precision problems, and even which instructions have accumulated errors that cause the precision problems of the operators with precision problems.

[0229] Attached Figure 8 Attached is a flow chart of a precision error positioning method provided in an embodiment of the present application. Figure 8 The method shown can be executed in parallel with the training operation or the reasoning operation. Figure 8 The method shown includes the following steps S710 to S780.

[0230] Step S710: The computing device obtains a pre-trained framework model, such as a pytorch model, a tensorflow model, a paddlepaddle model, an ONNX model, or a mindspore framework model.

[0231] In step S720, the computing device parses the pre-trained framework model and generates a DAG graph of the pre-trained framework model, thereby uniformly using a graph to express various types of pre-trained framework models. The DAG graph of the pre-trained framework model includes the upstream and downstream relationships of operators at each layer in the network.

[0232] Step S720 includes steps S721 to S723.

[0233] Step S721: The computing device parses the DAG of the pre-trained framework model to obtain a parsing result.

[0234] In step S722, the computing device generates a native intermediate representation (IR) of the pre-trained framework model based on the parsing result.

[0235] Step S723: The computing device generates a DAG of the pre-trained framework model based on the IR.

[0236] Step S730: The computing device extracts a subnet from the DAG.

[0237] Step S740: The computing device extracts the reference input data of the subnet.

[0238] Step S750: The computing device performs precision error positioning for multiple subnets in parallel.

[0239] Step S760: The computing device performs cumulative error positioning on the upper subnet.

[0240] Step S770 , the subsequent subnets of the operators with abnormal computing device accuracy perform accuracy error location.

[0241] Step S780: The computing device executes a training service or an inference service.

[0242] Step S780 may be performed in parallel with each of steps S710 to S770 .

[0243] This embodiment provides a method suitable for reasoning and training, which can delimit and locate the precision problem caused by the operator cumulative error. In addition, it can effectively locate the precision problem of the subsequent cascade of the precision abnormal operator.

[0244] In addition, a comprehensive analysis method of model accuracy error based on network topology structure automation is provided, thereby realizing the system-level parallel operator problem positioning from the original manual serial positioning.

[0245] In addition, a set of end-to-end real-time accuracy analysis system methods is provided, which can directly locate the accuracy problems of operator instructions from the accuracy problems of training models; and can directly locate the accuracy problems of operator instructions from the accuracy problems of inference models. It can realize real-time analysis and location of operator accuracy problems in training and inference processes.

[0246] In addition, a general precision error localization process suitable for reasoning and training is provided, which can distinguish error types and error locations, use a linearization algorithm to simplify the network topology, and use a search algorithm to improve search efficiency.

[0247] The present application also provides a precision error positioning device, as shown in the attached Fig. 9 As shown, the precision error positioning device 900 includes:

[0248] A determination module 910, configured to determine whether the accuracy of first output data of a first node in a tested network is abnormal;

[0249] An operation module 920, configured to input the reference input data corresponding to the first node into the first node, and operate the first node to obtain second output data of the first node;

[0250] The determination module 910 is further used to determine whether the accuracy of the second output data of the first node is normal;

[0251] An interception module 930, configured to intercept a first sub-network from the network under test;

[0252] The operation module 920 is further used to input the reference input data corresponding to the first sub-network into the first sub-network, and operate the first sub-network to obtain the output data of the first sub-network, where the first sub-network is an upper layer network located at the first node in the network under test;

[0253] The determination module 910 is further configured to determine that the first sub-network includes a cumulative error accuracy interval in response to determining that the accuracy of the output data of the first sub-network is abnormal.

[0254] In one possible implementation, the interception module 930 is used to determine the second subnetwork based on the network under test, the head node of the second subnetwork is the input node of the network under test, and the tail node of the second subnetwork is the first node; the reference input data corresponding to the second subnetwork is input into the second subnetwork, the second subnetwork is run, and the output data of the second subnetwork is obtained; based on the accuracy verification result of the output data of the second subnetwork, the first subnetwork is determined from the second subnetwork.

[0255] In a possible implementation, the second sub-network includes an intermediate node located between the head node and the tail node, and the operation module 920 is used to input the reference input data corresponding to the intermediate node into the intermediate node, and operate the sub-network from the intermediate node to the tail node in the second sub-network to obtain the output data of the second sub-network;

[0256] Determination module 910 is used to determine the head node of the first subnetwork based on the intermediate node of the second subnetwork and the tail node of the first subnetwork based on the tail node of the second subnetwork in response to determining that the accuracy of the output data of the second subnetwork is abnormal; or, in response to determining that the accuracy of the output data of the second subnetwork is normal, determine the tail node of the first subnetwork based on the intermediate node of the second subnetwork and the head node of the first subnetwork based on the head node of the second subnetwork.

[0257] In a possible implementation, the determination module 910 is configured to determine a backbone node sequence from an input node to a first node of the network under test; and determine a second sub-network from the backbone node sequence.

[0258] In a possible implementation, the determination module 910 is configured to intercept the parent node of the first node and the first node in the tested network to obtain the first sub-network.

[0259] In a possible implementation, the interception module 930 is further used to intercept a third sub-network from the network under test, input the reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network, and obtain the output data of the third sub-network, where the third sub-network is a lower layer network located at the first node in the network under test;

[0260] The determination module 910 is further configured to determine that there are nodes in the third sub-network with abnormal accuracy in response to determining that the accuracy of the output data of the third sub-network is abnormal.

[0261] In a possible implementation, the determination module 910 is further configured to determine that the first node has an accuracy anomaly in response to determining that the accuracy of the second output data of the first node is abnormal.

[0262] In one possible implementation, the network under test includes a second node and a third node, the third node is a child node of the second node, and a determination module 910 is used to determine the flow value of the third node based on the flow value of the second node and the number of child nodes of the second node; in response to determining that the flow value of the third node is the same as the flow value of the input node, the third node is added to the trunk node sequence.

[0263] Among them, the determination module 910, the operation module 920 and the interception module 930 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the determination module 910 is introduced below by taking the determination module 910 as an example. Similarly, the implementation of the operation module 920 and the interception module 930 can refer to the implementation of the determination module 910.

[0264] As an example of a software functional unit, the module 910 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance can be one or more. For example, the module 910 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Generally, a region can include multiple AZs.

[0265] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0266] As an example of a hardware functional unit, the determination module 910 may include at least one computing device, such as a server, etc. Alternatively, the determination module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0267] The multiple computing devices included in the determination module 910 may be distributed in the same region or in different regions. The multiple computing devices included in the determination module 910 may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the determination module 910 may be distributed in the same VPC or in multiple VPCs. The multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0268] It should be noted that, in other embodiments, the determination module 910 can be used to execute any step in the precision error positioning method, the operation module 920 can be used to execute any step in the precision error positioning method, and the interception module 930 can be used to execute any step in the precision error positioning method. The steps that the determination module 910, the operation module 920, and the interception module 930 are responsible for implementing can be specified as needed. The full functions of the precision error positioning device are realized by respectively implementing different steps in the precision error positioning method through the determination module 910, the operation module 920, and the interception module 930.

[0269] The present application also provides a computing device 100. Fig.10 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0270] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 104 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0271] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0272] The memory 106 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0273] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned determination module 910, the operation module 920, and the interception module 930, thereby implementing the precision error location method. That is, the memory 106 stores instructions for executing the precision error location method.

[0274] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0275] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0276] like Fig.11 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the accuracy error location method.

[0277] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the accuracy error location method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the accuracy error location method.

[0278] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the precision error positioning device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more modules in the determination module 910, the operation module 920 and the interception module 930.

[0279] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.12 A possible implementation is shown. Fig.12 As shown, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the determination module 910. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the operation module 920 and the interception module 930.

[0280] Fig.12 The connection method between the computing device clusters shown can be that considering that the precision error positioning method provided in the present application requires running nodes and intercepting sub-networks, it is considered that the functions implemented by the running module 920 and the intercepting module 930 are handed over to the computing device 100B for execution.

[0281] It should be understood that Fig.12The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0282] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.11 and Fig.12 Connection mode of computing device cluster. The difference is that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the accuracy error location method.

[0283] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the accuracy error location method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the accuracy error location method.

[0284] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the accuracy error location method.

[0285] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the accuracy error positioning method.

[0286] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0287] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0288] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe the specific order of the objects, and cannot be understood as indicating or implying relative importance. For example, the first subnetwork and the second subnetwork are used to distinguish different subnetworks, rather than to describe the specific order of the subnetworks, and cannot be understood as the first subnetwork being more important than the second subnetwork.

[0289] In the embodiments of the present application, unless otherwise specified, "at least one" means one or more, and "a plurality" means two or more. For example, a plurality of subnetworks means two or more subnetworks.

[0290] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state hard disk Solid State Disk (SSD)), etc.

[0291] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A precision error positioning method, characterized in that: The method comprises: In response to determining that the accuracy of first output data of a first node in the tested network is abnormal, inputting reference input data corresponding to the first node into the first node, and running the first node to obtain second output data of the first node; In response to determining that the accuracy of the second output data of the first node is normal, intercepting a first sub-network from the network under test, inputting reference input data corresponding to the first sub-network into the first sub-network, and running the first sub-network to obtain output data of the first sub-network, wherein the first sub-network is an upper-layer network located at the first node in the network under test; In response to determining that the accuracy of the output data of the first sub-network is abnormal, determining that the first sub-network includes a cumulative error accuracy interval.

2. The method according to claim 1, characterized in that The step of intercepting a first sub-network from the network under test comprises: Determine a second sub-network based on the network under test, wherein the head node of the second sub-network is the input node of the network under test, and the tail node of the second sub-network is the first node; Inputting the reference input data corresponding to the second sub-network into the second sub-network, running the second sub-network, and obtaining the output data of the second sub-network; Based on the accuracy verification result of the output data of the second sub-network, the first sub-network is determined from the second sub-network.

3. The method according to claim 2, characterized in that The second sub-network includes an intermediate node between the head node and the tail node, and the reference input data corresponding to the second sub-network is input into the second sub-network, and the second sub-network is run to obtain the output data of the second sub-network, including: Inputting the reference input data corresponding to the intermediate node into the intermediate node, running the subnetwork from the intermediate node to the tail node in the second subnetwork, and obtaining the output data of the second subnetwork; The determining the first sub-network from the second sub-network based on the accuracy check result of the output data of the second sub-network includes: In response to determining that the accuracy of the output data of the second subnetwork is abnormal, the head node of the first subnetwork is determined based on the intermediate node of the second subnetwork, and the tail node of the first subnetwork is determined based on the tail node of the second subnetwork; or, in response to determining that the accuracy of the output data of the second subnetwork is normal, the tail node of the first subnetwork is determined by the tail node of the second subnetwork, and the head node of the first subnetwork is determined based on the head node of the second subnetwork.

4. The method according to claim 2, characterized in that: The determining the second sub-network based on the network under test includes: Determine a backbone node sequence from an input node of the network under test to the first node; A second sub-network is determined from the backbone node sequence.

5. The method according to claim 1, characterized in that The step of intercepting a first sub-network from the network under test comprises: The parent node of the first node and the first node in the network under test are intercepted to obtain the first sub-network.

6. The method according to claim 1, characterized in that The method further comprises: Intercepting a third sub-network from the network under test, inputting reference input data corresponding to the third sub-network into the third sub-network, running the third sub-network, and obtaining output data of the third sub-network, wherein the third sub-network is a lower layer network located at the first node in the network under test; In response to determining that the accuracy of the output data of the third sub-network is abnormal, it is determined that a node with abnormal accuracy exists in the third sub-network.

7. The method according to claim 1, characterized in that The method further comprises: In response to determining that the precision of the second output data of the first node is abnormal, it is determined that the first node has a precision abnormality.

8. The method according to any one of claims 1 to 7, characterized in that The first node is the Mth layer network in the network under test, the first subnetwork is the Kth layer network to the Mth layer network in the network under test, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

9. The method according to any one of claims 1 to 7, characterized in that The network under test is a computing device cluster, which is used to execute a task sequence. The first node is a computing device or an AI accelerator card that executes the Mth task in the task sequence. The first subnetwork is a computing device or an AI accelerator card that executes the Kth task to the Mth task in the task sequence. M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

10. The method according to claim 4, characterized in that The network under test includes a second node and a third node, the third node is a child node of the second node, and determining a backbone node sequence from an input node of the network under test to the first node includes: determining a flow value of the third node based on the flow value of the second node and the number of child nodes of the second node; In response to determining that the flow value of the third node is the same as the flow value of the input node, the third node is added to the backbone node sequence.

11. A precision error positioning device, characterized in that: The device comprises: A determination module, configured to determine that the accuracy of first output data of a first node in a tested network is abnormal; an operating module, configured to input the reference input data corresponding to the first node into the first node, and operate the first node to obtain second output data of the first node; The determination module is further used to determine that the accuracy of the second output data of the first node is normal; An interception module, used for intercepting a first sub-network from the network under test; The operation module is further used to input the reference input data corresponding to the first sub-network into the first sub-network, and operate the first sub-network to obtain the output data of the first sub-network, wherein the first sub-network is an upper layer network located at the first node in the network under test; The determination module is further configured to determine that the first sub-network includes a cumulative error accuracy interval in response to determining that the accuracy of the output data of the first sub-network is abnormal.

12. The device according to claim 11, characterized in that The interception module is used to determine a second sub-network based on the network under test, the head node of the second sub-network is the input node of the network under test, and the tail node of the second sub-network is the first node; Inputting the reference input data corresponding to the second sub-network into the second sub-network, running the second sub-network, and obtaining the output data of the second sub-network; Based on the accuracy verification result of the output data of the second sub-network, the first sub-network is determined from the second sub-network.

13. The device according to claim 12, characterized in that The second sub-network includes an intermediate node located between the head node and the tail node, and the running module is used to input the reference input data corresponding to the intermediate node into the intermediate node, run the sub-network from the intermediate node to the tail node in the second sub-network, and obtain the output data of the second sub-network; The determining module is configured to, in response to determining that the accuracy of the output data of the second sub-network is abnormal, determine the head node of the first sub-network based on the intermediate nodes of the second sub-network, and determine the tail node of the first sub-network based on the tail node of the second sub-network; Alternatively, in response to determining that the accuracy of the output data of the second sub-network is normal, the tail node of the first sub-network is determined based on the intermediate nodes of the second sub-network, and the head node of the first sub-network is determined based on the head node of the second sub-network.

14. The device according to claim 12, characterized in that The determination module is used to determine a backbone node sequence from the input node of the network under test to the first node; and determine a second sub-network from the backbone node sequence.

15. The device according to claim 11, characterized in that The determination module is used to intercept the parent node of the first node and the first node in the tested network to obtain the first sub-network.

16. The device according to claim 11, characterized in that The interception module is further used to intercept a third sub-network from the network under test, input the reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network, and obtain the output data of the third sub-network, wherein the third sub-network is a lower layer network located at the first node in the network under test; The determination module is further configured to determine that there are nodes with abnormal accuracy in the third sub-network in response to determining that the accuracy of the output data of the third sub-network is abnormal.

17. The device according to claim 11, characterized in that The determination module is further configured to determine that the first node has an accuracy anomaly in response to determining that the accuracy of the second output data of the first node is abnormal.

18. The device according to claim 14, characterized in that The measured network includes a second node and a third node, the third node is a child node of the second node, and the determination module is used to determine the flow value of the third node based on the flow value of the second node and the number of child nodes of the second node; In response to determining that the flow value of the third node is the same as the flow value of the input node, the third node is added to the backbone node sequence.

19. A computing device cluster, characterized in that: The computing device cluster is a network under test, the computing device cluster includes at least one computing device, each computing device is a node in the network under test, and each computing device includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 10.

20. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 10.

Images