Precision error positioning method and apparatus

Abstract

The application provides a precision error positioning method and device, and belongs to the technical field of computers. In the embodiment of the application, in response to determining that the precision of the first output data of the first node in the measured network is abnormal, the reference input data corresponding to the first node is input to the first node, the first node is run, and the second output data of the first node is obtained; in response to determining that the precision of the second output data of the first node is normal, a first subnetwork is intercepted from the measured network, the reference input data corresponding to the first subnetwork is input to the first subnetwork, the first subnetwork is run, and the output data of the first subnetwork is obtained, and the first subnetwork is an upper network of the first node in the measured network; and in response to determining that the precision of the output data of the first subnetwork is abnormal, it is determined that the first subnetwork comprises an accumulated error precision interval. The embodiment of the application realizes accurate positioning of accumulated precision errors caused by an upper subnetwork.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a precision error positioning method and device. BACKGROUND

[0002] In a network, each layer subnetwork is connected with the next layer subnetwork, and the output data of the previous layer subnetwork is the input data of the next layer subnetwork. Therefore, if the output data of the previous layer subnetwork has precision error, the precision error of the previous layer subnetwork will usually be passed to the next layer subnetwork along with the network hierarchy, resulting in the precision error of the entire network being accumulated, and this accumulated error will make the precision error of the subsequent subnetworks become larger and larger.

[0003] Therefore, there is an urgent need to provide a solution that can locate the accumulated precision error caused by the upper layer subnetwork. SUMMARY

[0004] The embodiments of the present application provide a precision error positioning method and device, which can locate the accumulated precision error caused by the upper layer subnetwork. The technical solution is as follows.

[0005] In a first aspect, a precision error positioning method is provided, which includes: in response to determining that the precision of the first output data of the first node in the measured network is abnormal, inputting the reference input data corresponding to the first node into the first node, running the first node to obtain the second output data of the first node; in response to determining that the precision of the second output data of the first node is normal, intercepting a first subnetwork from the measured network, inputting the reference input data corresponding to the first subnetwork into the first subnetwork, running the first subnetwork to obtain the output data of the first subnetwork, and the first subnetwork is an upper layer network of the first node in the measured network; and in response to determining that the precision of the output data of the first subnetwork is abnormal, determining that the first subnetwork includes an accumulated error precision interval.

[0006] Since the reference input data can be understood as correct data, if the output data (second output data) generated by running the first node based on the correct data is abnormal in precision, and the output data (first output data) generated by running the first node based on the output data of the previous node of the first node is normal in precision, and the difference between the two times of running the first node is mainly that the input data used is different, it can be determined that the output data of the previous node of the first node causes the first node (second output data) to be abnormal in precision, and the previous node or the node before the previous node (upper subnetwork) of the first node has a cumulative error precision interval. By intercepting the upper subnetwork of the first node, the data flow relationship between the first node and the upper network of the first node is disconnected, the influence of the precision error of the output data of the upper network on the precision comparison of the first node is reduced, and then by inputting the corresponding reference input data to the upper subnetwork, if the output data of the upper network is abnormal in precision, it is determined that the upper subnetwork includes a cumulative error precision interval, so as to realize accurate positioning of the cumulative precision error caused by the upper subnetwork.

[0007] In a possible implementation, the intercepting the first subnetwork from the measured network comprises: determining a second subnetwork based on the measured network, wherein a head node of the second subnetwork is an input node of the measured network, and a tail node of the second subnetwork is the first node; inputting corresponding reference input data of the second subnetwork to the second subnetwork, running the second subnetwork, and obtaining output data of the second subnetwork; and determining the first subnetwork from the second subnetwork based on a precision checking result of the output data of the second subnetwork.

[0008] The first subnetwork is further determined from the second subnetwork based on the precision checking result of the second subnetwork, so as to narrow the range of the cumulative error precision interval from the second subnetwork to the first subnetwork, and improve the accuracy of the cumulative error precision interval.

[0009] In a possible implementation, the second subnetwork includes an intermediate node between the head node and the tail node, the inputting the corresponding reference input data of the second subnetwork to the second subnetwork, running the second subnetwork, and obtaining the output data of the second subnetwork comprises: inputting corresponding reference input data of the intermediate node to the intermediate node, running a subnetwork from the intermediate node to the tail node in the second subnetwork, and obtaining the output data of the second subnetwork; and the determining the first subnetwork from the second subnetwork based on the precision checking result of the output data of the second subnetwork comprises: in response to determining that the output data of the second subnetwork is abnormal in precision, determining the head node of the first subnetwork based on the intermediate node of the second subnetwork, and determining the tail node of the first subnetwork based on the tail node of the second subnetwork.

[0010] Since the check result of the first sub-network from the intermediate node to the tail node has precision anomaly, it indicates that the intermediate node to the tail node includes the accumulated error precision interval, by resetting the head node, it is equivalent to moving the head node to the position of the intermediate node, and then re-dividing the first sub-network from the new intermediate node to the tail node, thereby reducing the range of the searched accumulated error precision interval by half, excluding the range of the original head node to the intermediate node, and improving the accuracy of the accumulated error precision interval.

[0011] In a possible implementation, in response to determining that the accuracy of the output data of the second sub-network is normal, the tail node of the second sub-network determines the tail node of the first sub-network, and the head node of the first sub-network is determined based on the head node of the second sub-network.

[0012] Since the check result of the first sub-network has normal precision, it indicates that the accumulated error precision interval is not included in the intermediate node to the tail node, and therefore the intermediate node to the tail node can be excluded from the range of the accumulated error precision interval. Since it has been verified that the head node to the tail node includes the accumulated error precision interval, it can be determined that the accumulated error precision interval is located at the node next to the head node to the intermediate node. By resetting the tail node, it is equivalent to moving the tail node to the node next to the intermediate node, and then re-dividing the first sub-network from the head node to the new tail node, thereby reducing the range of the searched accumulated error precision interval by half, and improving the accuracy of the accumulated error precision interval.

[0013] In a possible implementation, the determining the second sub-network based on the measured network comprises: determining a main node sequence from the input node of the measured network to the first node; and determining the second sub-network from the main node sequence.

[0014] By determining the main node sequence, the determined main node sequence includes a complete sub-network, and the determined main node sequence is on the same straight line, and the main node sequence structure is linear without branches or redundant process nodes, thereby avoiding the process nodes being cut into the sub-network, which leads to the need to perform additional input reference data and compare the output results of the process nodes, thereby reducing the operation complexity of the input data and the comparison of the output results, and further improving the positioning error speed.

[0015] In a possible implementation, the intercepting the first sub-network from the measured network comprises: intercepting the parent node of the first node and the first node in the measured network to obtain the first sub-network.

[0016] By intercepting the parent node of the first node to the first node as the first sub-network, it is helpful to minimize the cumulative error precision interval as much as possible, so as to locate the position of the cumulative error more quickly. Because the parent node is adjacent to the node that appears the precision problem, it may be the node that directly affects the output result of the node that appears the precision problem, so starting from the parent node to investigate can effectively narrow the range of the problem.

[0017] In a possible implementation, the method further includes: intercepting a third sub-network from the measured network, inputting reference input data corresponding to the third sub-network into the third sub-network, running the third sub-network, and obtaining output data of the third sub-network, the third sub-network being a lower network of the first node in the measured network; and in response to determining that the precision of the output data of the third sub-network is abnormal, determining that the third sub-network has a node with precision abnormality.

[0018] In a possible implementation, the method further includes: in response to determining that the precision of the second output data of the first node is abnormal, determining that the first node has a precision abnormality.

[0019] Since the output data generated by the first node based on the reference input data (correct data) is still precision abnormal, it is determined that the first node itself has a precision abnormality. Based on this way, it can be more accurately determined that the problem occurs on which specific operator, and basically no debugging is required for the entire network, saving time and resources.

[0020] In a possible implementation, the first node is an Mth network in the measured network, and the first sub-network is a Kth network to the Mth network in the measured network, M being a positive integer greater than 1, and K being a positive integer less than or equal to M.

[0021] In a possible implementation, the measured network is a computing device cluster, the computing device cluster is used to execute a task sequence, the first node is a computing device or an AI accelerator card that executes an Mth task in the task sequence, and the first sub-network is a computing device or an AI accelerator card that executes a Kth task to the Mth task in the task sequence, M being a positive integer greater than 1, and K being a positive integer less than or equal to M.

[0022] In the above manner, it is applicable to the scenario of locating the cumulative error precision interval in the computing device cluster. For example, in the scenario of executing a task sequence in a distributed manner in the computing device cluster, one or more computing devices that have a precision problem when executing tasks are determined from the computing device cluster based on the result of each computing device in the computing device cluster executing a corresponding task.

[0023] In a possible implementation, the measured network comprises a second node and a third node, the third node being a child node of the second node, and the determining the backbone node sequence from an input node to the first node in the measured network comprises: determining a traffic value of the third node based on a traffic value of the second node and a number of child nodes of the second node; and in response to determining that the traffic value of the third node is the same as the traffic value of the input node, adding the third node to the backbone node sequence.

[0024] In a possible implementation, the first node comprises a first instruction and a second instruction before the first instruction, and after the determining that the first node has the precision abnormality, the method further comprises: inputting reference input data corresponding to the first instruction into the first instruction, executing the first instruction to obtain output data of the first instruction; in response to determining that the precision of the output data of the first instruction is normal, inputting reference input data corresponding to the second instruction into the second instruction, executing the second instruction to obtain output data of the second instruction; and in response to determining that the precision of the output data of the second instruction is abnormal, determining that the second instruction comprises a cumulative error precision interval.

[0025] In this way, an instruction-level cumulative error precision positioning means is provided, which helps to locate the operator with a precision problem and determine that the operator precision problem is caused by accumulation of several instructions.

[0026] In a possible implementation, the first output data of the first node, the second output data of the first node and the output data of the first sub-network are obtained by a first processor, and the reference input data corresponding to the first node, the reference input data corresponding to the first sub-network, the reference output data corresponding to the first node and the reference output data corresponding to the first sub-network are obtained by a second processor.

[0027] In a possible implementation, the determining the precision abnormality of the first output data of the first node in the measured network comprises: determining that a difference between the first output data of the first node and reference output data corresponding to the first node is greater than or equal to a threshold value; the determining the precision normality of the second output data of the first node comprises: determining that a difference between the second output data of the first node and the reference output data corresponding to the first node is less than or equal to the threshold value; and the determining the precision abnormality of the output data of the first sub-network comprises: determining that a difference between the output data of the first sub-network and reference output data corresponding to the first sub-network is greater than or equal to the threshold value.

[0028] In a possible implementation, the first node is an Mth operator in the network under test, and the first sub-network is an Kth operator to the Mth operator in the network under test, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0029] In a possible implementation, the first node is an Mth instruction in the network under test, and the first sub-network is an Kth instruction to the Mth instruction in the network under test, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0030] In a possible implementation, the intercepting the first sub-network from the network under test comprises: determining a topology of the first sub-network based on a directed acyclic graph of the network under test; and determining the first sub-network based on the topology of the first sub-network.

[0031] A directed acyclic graph (DAG) can clearly reflect the calculation flow and dependency relationship in the network under test. By analyzing the DAG, the data transmission and calculation order between nodes can be understood, thereby helping to more accurately determine the nodes included in the first sub-network.

[0032] In a possible implementation, the intercepting the first sub-network from the network under test comprises: performing the intercepting the first sub-network and the inference or training in parallel in a process of running inference or training based on the network under test. This way supports the intercepting the first sub-network and the inference or training to be performed at the same time, fully utilizes the computing resources, and accelerates the execution speed of the whole process. In addition, the precision abnormality can be detected in real time during the inference or training process, which is more timely.

[0033] In a second aspect, an accuracy error positioning apparatus is provided, and the apparatus comprises:

[0034] A determining module configured to determine that the accuracy of first output data of a first node in a network under test is abnormal.

[0035] A running module configured to input reference input data corresponding to the first node to the first node, and run the first node to obtain second output data of the first node.

[0036] The determining module is further configured to determine that the accuracy of the second output data of the first node is normal.

[0037] An intercepting module configured to intercept a first sub-network from the network under test.

[0038] The running module is further configured to input the reference input data corresponding to the first sub-network into the first sub-network, and run the first sub-network to obtain output data of the first sub-network, the first sub-network being an upper network of the first node in the network under test.

[0039] The determining module is further configured to determine that the first sub-network includes the cumulative error precision interval, in response to determining that the accuracy of the output data of the first sub-network is abnormal.

[0040] In a possible implementation, the intercepting module is configured to determine a second sub-network based on the network under test, the head node of the second sub-network being the input node of the network under test, and the tail node of the second sub-network being the first node; input reference input data corresponding to the second sub-network into the second sub-network, and run the second sub-network to obtain output data of the second sub-network; and determine the first sub-network from the second sub-network based on the accuracy check result of the output data of the second sub-network.

[0041] In a possible implementation, the second sub-network includes an intermediate node between the head node and the tail node, and the running module is configured to input reference input data corresponding to the intermediate node into the intermediate node, and run a sub-network from the intermediate node to the tail node in the second sub-network to obtain the output data of the second sub-network.

[0042] The determining module is configured to determine the head node of the first sub-network based on the intermediate node of the second sub-network, and determine the tail node of the first sub-network based on the tail node of the second sub-network, in response to determining that the accuracy of the output data of the second sub-network is abnormal; or determine the tail node of the first sub-network based on the intermediate node of the second sub-network, and determine the head node of the first sub-network based on the head node of the second sub-network, in response to determining that the accuracy of the output data of the second sub-network is normal.

[0043] In a possible implementation, the determining module is configured to determine a backbone node sequence from the input node of the network under test to the first node; and determine the second sub-network from the backbone node sequence.

[0044] In a possible implementation, the determining module is configured to intercept a parent node of the first node and the first node in the network under test to obtain the first sub-network.

[0045] In a possible implementation, the intercepting module is further configured to intercept a third sub-network from the network under test, input reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network, and obtain output data of the third sub-network, the third sub-network being a lower network of the first node in the network under test.

[0046] The determining module is further configured to determine that the third sub-network has a precision abnormality in response to determining that the output data of the third sub-network has a precision abnormality.

[0047] In a possible implementation, the determining module is further configured to determine that the first node has a precision abnormality in response to determining that the second output data of the first node has a precision abnormality.

[0048] In a possible implementation, the network under test includes a second node and a third node, the third node being a child node of the second node, and the determining module is configured to determine a traffic value of the third node based on a traffic value of the second node and a number of child nodes of the second node, and add the third node to the sequence of backbone nodes in response to determining that the traffic value of the third node is the same as the traffic value of the input node.

[0049] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the precision error locating method according to the first aspect or any implementation manner of the first aspect.

[0050] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores instructions. The instructions instruct the computing device or the computing device cluster to execute the precision error locating method according to the first aspect or any implementation manner of the first aspect.

[0051] In a fifth aspect, the present application provides a computer program product including instructions, and the instructions cause the computing device or the computing device cluster to execute the precision error locating method according to the first aspect or any implementation manner of the first aspect when the computer program product is executed on the computing device or the computing device cluster.

[0052] On the basis of the implementation manners of the aspects provided in the present application, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a flowchart of a precision error positioning method provided by an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of precision error positioning provided by an embodiment of the present application;

[0055] Figure 3 is a flowchart of a precision error positioning method provided by an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of determining a backbone node sequence provided by an embodiment of the present application;

[0057] Figure 5 is a schematic diagram of searching a cumulative error precision interval by using a binary search provided by an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of determining a cumulative error precision interval by using a ternary method provided by an embodiment of the present application;

[0059] Figure 7 is a flowchart of a precision error positioning method provided by an embodiment of the present application;

[0060] Figure 8 is a flowchart of a precision error positioning method provided by an embodiment of the present application;

[0061] Figure 9 is a structural schematic diagram of a precision error positioning apparatus provided by an embodiment of the present application;

[0062] Figure 10 is a structural schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0063] Figure 11 is a structural schematic diagram of another computing device cluster provided by an embodiment of the present application;

[0064] Figure 12 is a structural schematic diagram of still another computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0066] Some terms and concepts related to the embodiments of the present application will be explained below.

[0067] (1) Precision cumulative error

[0068] The precision accumulation error refers to a precision problem caused by the accumulation of precision errors of multiple nodes (e.g., multiple layers of operators) and exceeding a set threshold of a precision error of a node downstream of the multiple nodes (e.g., a layer of operators subsequent to the multiple layers of operators).

[0069] (2) Artificial Intelligence (AI) accelerator card

[0070] An AI accelerator card is a hardware used for performing AI computing tasks. The main function of an AI accelerator card is to provide efficient computing and storage capabilities to support large-scale deep learning model training and inference. For example, types of AI accelerator cards include Tensor Processing Units (TPUs), Graphics Processing Units (GPUs), Neural Processing Units (NPUs), Ascend AI processors, or Cambrian AI processors, etc.

[0071] FIG. 1 is a flowchart of a precision error positioning method according to an example embodiment of the present disclosure. Figure 1 FIG. 1 is a flowchart of a precision error positioning method according to an example embodiment of the present disclosure. Figure 1 The method shown in FIG. 1 can be performed by a computing device cluster. For example, the method shown in FIG. 1 is performed by one computing device in the computing device cluster, and each computing device in the computing device cluster is a node in the network under test; or the method shown in FIG. 1 is cooperatively performed by different computing devices in the computing device cluster. Figure 1 Figure 1 The method shown in FIG. 1 can be performed by a computing device cluster. For example, the method shown in FIG. 1 is performed by one computing device in the computing device cluster, and each computing device in the computing device cluster is a node in the network under test; or the method shown in FIG. 1 is cooperatively performed by different computing devices in the computing device cluster.

[0072] At step S100, the computing device obtains output data of each node of the network under test and reference data corresponding to each node.

[0073] The network under test includes multiple nodes. In one possible implementation, the network topology of the network under test has multiple layers. For example, the network under test includes an input layer, an intermediate layer, and an output layer. Each layer of the network under test has one or more nodes. The nodes of each layer are connected to the nodes of the next layer. The output data of the nodes of each layer is the input data of the nodes of the next layer. In an example embodiment, the network topology of the network under test is a directed acyclic graph.

[0074] ​In an example embodiment, the network under test is a machine learning model. For example, the network under test is a deep learning model. For example, the network under test is a convolutional neural network. Optionally, a node in the network under test is an operator. The granularity of the operator can be large or small. A large granularity operator is, for example, an independent network. For example, the network under test includes a Long Short-Term Memory (LSTM) operator. The LSTM operator itself is a network. A small granularity operator is, for example, an operation instruction or a combination of operation instructions. For example, a node (e.g., an operator) in the network under test includes at least one of a matrix operation instruction, a vector operation instruction, or a scalar operation instruction. For example, a node (e.g., an operator) in the network under test is a convolution operator, a fully connected operator, an activation function operator, a pooling operator, a normalization operator, a loss function operator, a deconvolution operator, or a depthwise separable convolution operator. For another example, each node in the network under test is an interface, which is encapsulated from a plurality of operation instructions. For example, the network under test is a model trained based on a framework such as PyTorch, ONNX, Caffe, TensorFlow, PaddlePaddle, or MindSpore.

[0075] In a possible implementation, the output data of each node of the network under test is a tensor. A tensor is used to represent multi-dimensional data. A tensor is, for example, a scalar (0-dimensional), a vector (1-dimensional), a matrix (2-dimensional), or an array of higher dimensions. A tensor can also be referred to as raw data. In the field of AI, the input data and the output data of a network are usually represented in the form of a tensor.

[0076] The reference data is also referred to as golden data or benchmark data. The reference data functions as a benchmark, which is used as a reference object when accuracy is compared. The reference data includes reference input data and reference output data. The reference output data is used to compare with the output data of the network under test. For example, the reference output data corresponding to the output layer node indicates the type of an image. The reference output data corresponding to the intermediate layer node indicates the features of an image.

[0077] In a possible implementation, the output data of each node of the measured network is generated by the first AI acceleration card. The benchmark data is generated by the second AI acceleration card. For example, the computing device includes the first AI acceleration card and the second AI acceleration card. The computing device runs the measured network through the first AI acceleration card to obtain the output data of each node of the measured network. The computing device runs the benchmark network through the second AI acceleration card to obtain the output data of each node of the benchmark network. For another example, the computing device cluster includes a first computing device and a second computing device, the first computing device includes the first AI acceleration card, and the second computing device includes the second AI acceleration card. The first computing device runs the measured network through the first AI acceleration card to obtain the output data of each node of the measured network. The second computing device runs the benchmark network through the second AI acceleration card to obtain the output data of each node of the benchmark network.

[0078] In a possible implementation, the benchmark network and the measured network are the same. For example, the benchmark network and the measured network have the same network structure, for example, the benchmark network and the measured network have the same number of layers; for another example, the nodes at the same layer in the benchmark network and the measured network have the same type; for another example, the nodes at the same layer in the benchmark network and the measured network have the same number of child nodes. Since the benchmark network and the measured network have the same network structure, the error caused by the difference in network structure is excluded in subsequent accuracy comparison, which is more reliable.

[0079] In another possible implementation, the benchmark network is model converted to obtain the measured network. For example, the data format of the model parameters (such as weight parameters, bias terms, and other model parameters) in the benchmark network is converted to a data format suitable for the first AI acceleration card, and data type matching, shape conversion, and the like are performed to obtain the measured network.

[0080] In a possible implementation, the measured network and the benchmark network are both image recognition models. The computing device inputs the same image into the measured network and the benchmark network respectively, and identifies the image through the measured network and the benchmark network respectively, to obtain the output data of each node by the measured network and the benchmark data corresponding to each node by the benchmark network. For example, the output data of an operator obtained based on a GPU or a central processing unit (CPU) running a model is taken as benchmark output data, and the output data of an operator generated by running a model through a Ascend AI processor is compared with the benchmark output data.

[0081] In a possible implementation, the computing device dumps the output data of a node in the benchmark network each time the output data of the node is obtained during the running of the benchmark network, for example, saves the output data of the node into a file, so that the output data of the node is obtained by accessing the saved file subsequently.

[0082] In step S110, the computing device compares the output data of each node in the measured network with the corresponding benchmark data of each node in terms of accuracy, so as to determine the node in the measured network that has accuracy anomaly.

[0083] For example, the computing device compares the first output data of the first node in the measured network with the corresponding benchmark output data of the first node in terms of accuracy, obtains the gap between the first output data and the corresponding benchmark output data of the first node, and obtains the first gap. The computing device determines whether the first gap is greater than a set threshold. If the first gap is greater than the set threshold, it is determined that the first node has accuracy anomaly (or the first node has accuracy problem), and step S120 is performed. If the first gap is less than or equal to the set threshold, it is determined that the first node has normal accuracy. The first node is any node in the measured network. The accuracy comparison process for other nodes can refer to the accuracy comparison process for the first node.

[0084] In a possible implementation, the accuracy comparison process is a tensor comparison process. For example, the output data of the first node is a tensor, and the corresponding benchmark output data of the first node is also a tensor. The computing device compares the two tensors, determines the gap between the two tensors, and determines whether the gap is greater than a set threshold.

[0085] The ways of determining the gap between two tensors include at least one of an absolute error, a relative error, a K2 divergence, a cosine similarity, an absolute relative error, a maximum relative error, or a mean relative error. The cosine similarity is a measurement method of measuring the difference in direction and size between two vectors. When two tensors are regarded as vectors, the cosine similarity can be used to determine the gap between the two tensors. The absolute error is a way of representing the difference between two tensors, and the absolute error is measured by calculating the absolute value of the difference between corresponding elements in the two tensors. The relative error refers to determining the gap between two tensors using the ratio of the absolute value of the difference between the elements of the two tensors to the average of the corresponding elements in the two tensors. The K2 divergence is to regard both tensors as probability distributions, and the K2 divergence is used to calculate the difference between the two tensors. The absolute relative error is an improved measurement method of the relative error, and the absolute relative error determines the gap between two tensors using the ratio of the absolute value of the difference between the elements of the two tensors to the absolute value of the corresponding elements in the two tensors. The maximum relative error is the maximum value of the relative error of all elements in the two tensors. The mean relative error is the average value of the relative error of all elements in the two tensors.

[0086] In a possible implementation, the computing device traverses the output data of each node in the measured network; for the i-th node in the measured network, the computing device performs precision comparison on the output data of the i-th node in the measured network and the reference output data of the i-th node in the reference network; if the precision comparison result of the i-th node indicates an abnormality, the cumulative error precision interval is located for the upper subnet of the i-th node; if the precision comparison result of the i-th node indicates normality, the next node of the i-th node is continued to be traversed. i is a positive integer.

[0087] In a possible implementation, the computing device takes the output data generated by the measured network running the second AI acceleration card as the reference output data, and compares the output data generated by the first AI acceleration card with the reference output data, so as to determine whether the output data generated by the measured network running the second AI acceleration card has a precision problem.

[0088] In step S120, in response to determining that the precision of the first output data of the first node in the measured network is abnormal, the computing device inputs the reference input data corresponding to the first node into the first node, runs the first node, and obtains second output data of the first node.

[0089] The embodiment relates to a process in which a same node is run multiple times, and data input to the node each time the node is run can be different, resulting in output data obtained each time the node is run can be different. In order to distinguish the output data obtained by running the node different times, "first output data" is used to describe output data obtained by running the first node based on output data of a node above the first node, and "second output data" is used to describe output data obtained by running the first node based on reference input data.

[0090] In a possible implementation, the computing device determines that the accuracy of the first output data of the first node is abnormal, and then the computing device intercepts the first node from the measured network, thereby breaking the data flow relationship between the first node and the upper network of the first node, and reducing the influence of the accuracy error of the output data of the upper network on the accuracy comparison of the first node; the computing device obtains reference input data corresponding to the first node from the reference network data, and runs the first node to process the reference input data, to obtain second output data output by the first node. Since the reference input data can be understood as correct data, if the output data (the second output data) generated by running the first node based on the correct data is abnormal in accuracy, and the output data (the first output data) generated by running the first node based on the output data of the node above the first node is normal in accuracy, and the difference between the two times of running the first node is mainly that the input data used is different, it can be determined that the output data of the node above the first node causes the accuracy of the first node (the second output data) to be abnormal, and the node above the first node or the node before the node above the first node (the upper subnetwork) has accumulated error accuracy intervals.

[0091] In step S140, in response to determining that the accuracy of the second output data of the first node is normal, the computing device intercepts the first subnetwork from the measured network, and the computing device inputs reference input data corresponding to the first subnetwork to the first subnetwork, and runs the first subnetwork to obtain output data of the first subnetwork.

[0092] For example, the computing device obtains a gap between the second output data of the first node and reference output data corresponding to the first node, to obtain a second gap. The computing device compares the second gap with a set threshold value. If the second gap is less than or equal to the set threshold value, the computing device determines that the accuracy of the second output data is normal. If the second gap is greater than the set threshold value, the computing device determines that the accuracy of the second output data is abnormal.

[0093] The first subnetwork is the network above the first node in the network under test. For example, if the first node is the Mth node, the first subnetwork is the network consisting of the Kth to Mth nodes in the network under test. Similarly, if the first node is the Mth layer network, the first subnetwork is the network consisting of the Kth to Mth layers in the network under test. Or, if the first node is the Mth instruction, the first subnetwork is the network consisting of the Kth to Mth instructions in the network under test. M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0094] Since the accuracy issue occurred when running the first node based on the output data of the previous node, but not when running the first node based on correct input data (reference input data), the possibility of an accuracy problem in the first node itself can be ruled out. Therefore, the root cause of the accuracy problem lies in the upper-level sub-network of the first node (the first sub-network). In other words, the type of accuracy anomaly in the output data (first output data) generated during the first run of the first node is due to cumulative error. Accuracy analysis of the first sub-network can be performed to more accurately pinpoint the accuracy problem.

[0095] In one possible implementation, the computing device acquires the directed acyclic graph (DAG) of the network under test; the computing device determines the network topology of the network under test based on the DAG; the computing device determines the upstream and downstream relationships between nodes based on the network topology of the network under test; and the computing device determines the upstream node of the first node based on the upstream and downstream relationships between nodes, thereby determining the first sub-network.

[0096] Step S160: In response to determining that the output data of the first sub-network is abnormal in precision, the first sub-network is determined to be within the cumulative error precision range.

[0097] The computing device obtains the baseline output data corresponding to the first sub-network from the baseline network data; the baseline output data corresponding to the first sub-network is, for example, the baseline output data corresponding to the first node. For example, the first sub-network consists of nodes K to M, and the baseline output data corresponding to the first sub-network is, for example, the baseline output data corresponding to the Mth node. The computing device obtains the difference between the output data of the first sub-network and the baseline output data corresponding to the first sub-network, and determines whether the difference is greater than a set threshold. If the difference is greater than the set threshold, the computing device determines that the accuracy of the output data of the first sub-network is abnormal.

[0098] For example, please refer to the appendix. Figure 2 The network under test is, for example, Figure 2 In network 300, the first node is, for example, attached. Figure 2 The Mth layer node, i.e., the attached Figure 2Node 301, the first subnetwork is, for example, attached Figure 3 Network 310 refers to the network from node 302 to node 301 in layer K. In other words, the error accuracy range is from node 302 to node 301. Node 302 is the starting point for accumulated error, and node 301 is the first node to detect the accuracy error.

[0099] Optionally, if the first sub-network is determined to be within the cumulative error accuracy range, the computing device replaces or repairs nodes in the first sub-network that have accuracy errors, thereby improving the accuracy of the first sub-network. Alternatively, the computing device merges some operators in the first sub-network to improve accuracy.

[0100] The method provided in this embodiment, when it is determined that the output data of a node is abnormally accurate, replaces the output data of the previous node with the reference input data. If the accuracy of the newly obtained output data is abnormal, it can be determined that the previous accuracy of the output data of the node was due to accumulated error. By extracting the upper sub-network of the node and inputting the corresponding reference input data into the upper sub-network, if the accuracy of the output data of the upper network is abnormal, it is determined that the upper sub-network includes the accuracy range of accumulated error, thereby achieving accurate positioning of accumulated accuracy error.

[0101] Optional, attached Figure 1 The method shown is applied to scenarios involving locating the accuracy interval of cumulative error in deep learning models. Alternatively, see attached... Figure 1 The method shown is applied to scenarios where at least one of the following is located in a deep learning model: an operator with precision error, a command with precision error, or an interface with precision error. Alternatively, see attached... Figure 1 The method shown is for scenarios involving accuracy error localization during inference or training processes using deep learning models. For example, see attached... Figure 1 The method described is applied to compare the accuracy of output data obtained by running the same deep learning model on different hardware (such as different AI accelerator cards). For example, the first node is the Mth operator in the network under test, and the first sub-network consists of the Kth to Mth operators in the network under test, where M is a positive integer greater than 1 and K is a positive integer less than or equal to M. Alternatively, the first node can be the Mth layer of the network under test, and the first sub-network consists of the Kth to Mth layers of the network under test, where M is a positive integer greater than 1 and K is a positive integer less than or equal to M. In this way, it is possible to pinpoint which operator in the entire network has an accuracy problem, or which operators' accumulated errors are causing the accuracy problem.

[0102] Optional, attached Figure 1The method is applied to a scenario of locating a cumulative error precision interval in a computing device cluster. For example, in a scenario of performing a task sequence in a distributed manner in a computing device cluster, one or more computing devices having precision problems are determined from the computing device cluster based on results of performing corresponding tasks by each computing device in the computing device cluster.

[0103] In another example scenario, the network under test is a computing device cluster, which includes a plurality of computing devices, such as servers or terminals. Each computing device includes one or more AI acceleration cards. A node in the network under test is, for example, a computing device in the computing device cluster or an AI acceleration card. The computing device cluster is used to perform a task sequence. The task sequence includes a plurality of tasks, each of which is performed by a computing device or an AI acceleration card. For example, the task sequence is a model training task, and each task in the task sequence is a parameter update of a model. The first node is a computing device or an AI acceleration card performing an Mth task in the task sequence, and the first sub-network is a computing device or an AI acceleration card performing a Kth task to an Mth task in the task sequence, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0104] Optionally, the method further includes: Figure 1 The method is applied to a scenario of locating an instruction interval having a cumulative error precision interval in an operator. For example, the first node is a Mth instruction in the network under test, and the first sub-network is a Kth instruction to a Mth instruction in the network under test, where M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

[0105] In one possible implementation, after determining that the output data precision of the first node is abnormal, the computing device further performs precision error positioning on a subsequent cascaded sub-network of the first node. The process of performing precision positioning on the subsequent cascaded sub-network includes steps one to three, for example.

[0106] Step one, the computing device intercepts a third sub-network from the network under test.

[0107] The third sub-network is a lower network of the first node in the network under test. For example, the computing device determines each downstream node of the first node in the network under test based on the DAG of the network under test, and takes each downstream node of the first node as the third sub-network. For example, the network under test includes N nodes, the first node is an Mth node in the network under test, and the third sub-network is each downstream node of the Mth node in the network under test, for example, the head node of the third sub-network is an (M+1)th node in the network under test, and the tail node of the third sub-network is an Nth node in the network under test. For example, refer to FIG. 3. Figure 2 The third sub-network is, for example, the sub-network 320 in FIG. 3. Figure 2 ​

[0108] Since the third sub-network is intercepted, the data flow relationship between the third sub-network and the first node and the upper-layer sub-network of the first node is disconnected, so that the third sub-network becomes a relatively independent sub-network. Therefore, when the precision error analysis of the third sub-network is performed subsequently, the output data of the node with the precision problem in the first node and the upper-layer sub-network of the first node reduces the interference on the precision error positioning of the third sub-network, and the accuracy of the precision error positioning of the third sub-network is improved.

[0109] In step two, the computing device inputs the reference input data corresponding to the third sub-network into the third sub-network; and the computing device runs the third sub-network to obtain the output data of the third sub-network.

[0110] For example, the head node of the third sub-network is the (M+1)th node in the measured network, and the computing device inputs the reference input data corresponding to the (M+1)th node into the head node (the (M+1)th node in the measured network) of the third sub-network. Since the input data of the third sub-network is replaced from the output data of the first node to the reference input data, the third sub-network itself can be detected whether there is a precision problem by using the more accurate input data of the reference input data.

[0111] In step three, in response to determining that the output data of the third sub-network has a precision abnormality, a node with a precision abnormality in the third sub-network is determined.

[0112] For example, the difference between the output data of the third sub-network and the reference output data corresponding to the third sub-network (such as the output data corresponding to the last node in the measured network) is obtained to obtain a third difference. The computing device judges whether the third difference is greater than a set threshold value. If the third difference is greater than the set threshold value, the computing device determines that the output data of the third sub-network has a precision abnormality; in addition, if the third difference is less than or equal to the set threshold value, the computing device determines that the output data of the third sub-network has a normal precision.

[0113] By providing the above precision abnormality positioning method for the third sub-network, considering that the output data of the node with the precision problem in the measured network will flow into the subsequent cascaded sub-network, which will affect the precision of the nodes in the subsequent cascaded sub-network, and the precision error of the subsequent cascaded sub-network of the node with the precision problem cannot be accurately positioned, the subsequent cascaded sub-network of the node with the precision problem is intercepted, so that the data flow relationship between the subsequent cascaded sub-network and the node with the precision problem is disconnected, and the input data of the subsequent cascaded sub-network is replaced by relatively correct input data (reference input data). In the case that all input data are reference input data, if the difference between the output data of the third sub-network and the reference output data is greater than the threshold value, it indicates that the third sub-network itself has a precision problem, and thus the node with the precision abnormality in the third sub-network can be accurately determined.

[0114] Appendix Figure 1 The method described focuses on the case where the accuracy of the output data (second output data) generated after rerunning the first node based on the reference input data is normal. In other embodiments, if the accuracy of the output data (second output data) generated after rerunning the first node based on the reference input data is abnormal, the computing device can determine that the first node itself has an accuracy abnormality. In other words, the error type of the first node is a single-node error. Alternatively, it can be understood that the cumulative error interval extends from the beginning to the end of the first node.

[0115] Optionally, if the computing device determines that there is an accuracy anomaly in the first node, it further locates the instruction with the accuracy anomaly within the first node. For example, if the first node includes a first instruction and a second instruction preceding the first instruction, the computing device inputs the reference input data corresponding to the first instruction into the first instruction, executes the first instruction, and obtains the output data of the first instruction. In response to determining that the accuracy of the output data of the first instruction is normal, it inputs the reference input data corresponding to the second instruction into the second instruction, executes the second instruction, and obtains the output data of the second instruction. In response to determining that the accuracy of the output data of the second instruction is abnormal, it determines that the second instruction includes the cumulative error accuracy range. In this way, a means of locating the cumulative error accuracy at the instruction level is provided, which helps to locate which instructions caused the operator accuracy problem.

[0116] Appendix Figure 3 This is a flowchart of a precision error positioning method provided in an embodiment of this application. Figure 3 The method shown is for the attached Figure 1 The flowchart above shows the overall process for positioning within the cumulative error accuracy range, supplemented by the error inherent in each node. Figure 3 The method shown supports outputting the accuracy error of all nodes in the network under test. (Appendix) Figure 3 In the method shown, the M-th layer node is attached. Figure 1 A specific example of the first node in the method shown is attached. Figure 3 In the method shown, the sub-networks from layer K to layer M are attached. Figure 3 A specific example of the first subnetwork in the method shown.

[0117] Step S210: The computing device acquires input data, runs the network under test based on the input data to generate network under test data, and runs the benchmark network based on the input data to generate benchmark network data.

[0118] The network under test (DUT) data includes the output data of each node in each layer of the DUT. The benchmark network data includes the output data of each node in each layer of the benchmark network. In one possible implementation, the DUT and the benchmark network are the same. For example, the DUT and the benchmark network are the same neural network model, which is run on different AI accelerator cards to obtain the DUT and benchmark network data, respectively.

[0119] In one possible implementation, the computing device includes a first processor and a second processor. Accordingly, step S210 includes steps S211 and S212.

[0120] In step S211, the computing device runs the network under test based on the input data through the first processor and saves the network under test data generated by each node of the network under test.

[0121] In step S212, the computing device runs the benchmark network based on the input data through the second processor and saves the benchmark network data generated by each node in the benchmark network.

[0122] The input data used in steps S211 and S212 can be the same, for example, the network under test and the reference network are both image recognition models, and the input data used in steps S211 and S212 is the same image. Alternatively, the input data used in steps S211 and S212 can be the same tensor. Because the input data is the same, the difference in the output data can be used to measure the accuracy error of the operators in the network.

[0123] For example, the first processor is an NPU or Ascend AI processor. The second processor is a GPU or CPU. The baseline network (the original network) is a framework network such as Caffe, TensorFlow, or ONNX. The baseline network is compiled into an offline model supported by the first processor using tensors. The offline model is a concrete example of the network under test. Additionally, a fusion rule mapping file is output, recording the mapping relationship between the third-party framework and the nodes of the network under test. The first processor loads the offline model (network under test) and input data via ACL, performs inference based on the offline model and input data, and dumps the output data of each layer. The second processor runs the baseline network, thus dumping the output data of each layer. Because the default data type of GPUs or CPUs is fp32 (32-bit floating-point type), while the data type on the Ascend AI processor is fp16 (16-bit floating-point type), the difference in the representation width of floating-point data leads to differences in the precision of the output data of each layer of the network for different processors. Optionally, the output data of each layer node dumped by the GPU or CPU is stored in NumPy format (npy).

[0124] Step S211 and step S212 can be executed sequentially. For example, step S211 can be executed first, and then step S212 can be executed. Alternatively, step S212 can be executed first, and then step S211 can be executed. In some other embodiments, step S211 and step S212 can also be executed in parallel, that is, step S211 and step S212 can be executed simultaneously.

[0125] In step S220, the computing device compares the output data of each layer node in the measured network data with the output data of each layer node in the benchmark network data, and determines whether the output data of the Mth layer node in the measured network has an accuracy abnormality.

[0126] In one possible implementation, the output data of each layer node in the measured network data is a tensor (a tensor is also referred to as raw data), and the output data of each layer node in the benchmark network data is also a tensor. The output tensor of the ith layer node in the measured network data is compared with the output tensor of the ith layer node in the benchmark network data. The gap between the output tensor of the ith layer node in the measured network data and the output tensor of the ith layer node in the benchmark network data is obtained. If the gap between the two output tensors is greater than or equal to a gap threshold, it is determined that the output data of the ith layer node has an accuracy abnormality. If the gap between the two output tensors is less than or equal to the gap threshold, it is determined that the output data of the ith layer node is normal. i is a positive integer greater than or equal to 1 and less than or equal to M.

[0127] The manner of determining the gap between the two output tensors is, for example, an absolute error, a relative error, a K2 divergence, a cosine similarity, an absolute relative error, or an average relative error. The gap threshold is determined based on experience or a business scenario to which the measured network is applied.

[0128] In one possible implementation, the computing device starts from the first layer node in the measured network data, traverses each layer node in the measured network data, and compares the output data of the ith layer node in the measured network data with the output data of the ith layer node in the benchmark network data. If the output data of the ith layer node in the measured network has an accuracy abnormality, step S240 is performed for the ith layer in the measured network. In this case, i is a specific example of M, and the Mth layer is equivalent to a first-level accuracy abnormality point. If the output data of the ith layer node in the measured network is normal, the output data of the (i+1)th layer node in the measured network is continuously traversed until the output data of the last layer node in the measured network is traversed.

[0129] Step S240: For the M-th layer node in the network data under test, the computing device obtains the reference input data corresponding to the M-th layer node from the reference network data, inputs the reference input data corresponding to the M-th layer node into the M-th layer node, and re-runs the M-th layer node through the first processor to process the reference input data corresponding to the M-th layer node, thereby obtaining the regenerated output data (second output data) of the M-th layer node. The computing device obtains the reference output data of the M-th layer node from the reference network data; the computing device performs a precision comparison between the regenerated output data of the M-th layer node and the reference output data of the M-th layer.

[0130] For example, a node in layer M is a single node. In other words, layer M is a single-node network. The reference input data corresponding to layer M is, for example, the output data generated by the (M-1)th layer node in the reference network after the second processor runs the reference network.

[0131] If the output data of the regenerated M-th layer node has normal precision, the computing device can determine that the reason for the abnormal precision of the output data of the M-th layer node is the cumulative error. That is, the abnormal precision of the output data of the first generated M-th layer node is caused by the cumulative error precision range of the upper sub-network of the M-th layer node, rather than the M-th layer node itself causing the abnormal precision of the output data of the first generated M-th layer node. In this case, step S260 is further executed.

[0132] If the output data of the regenerated layer M node has abnormal precision, the computing device determines that the layer M node itself has a cumulative precision error. In other words, the computing device determines that the precision error type is a single-node error, and in this case, the computing device further executes step S250.

[0133] Step S250: The computing device records the information of the M-th layer node.

[0134] Step S260: The computing device performs accuracy error positioning on the subsequent cascaded subnets of the M-th layer node.

[0135] Step S260 includes the following steps S261 to S263.

[0136] Step S261: The computing device extracts the sub-networks from layer K to layer M in the network under test.

[0137] In the process of selecting the K layer, the computing device first selects the parent node (the (M-1) layer node) of the M layer in the measured network as the K layer, inputs the (M-1) layer corresponding reference input data to the (M-1) layer node, and if the output data accuracy of the M layer node is abnormal, the computing device determines that the (M-1) layer node has accuracy abnormality. If the output data accuracy of the M layer node is normal, the computing device determines that the accuracy abnormality of the M layer node is caused by the upper subnet of the (M-1) layer node.

[0138] In step S262, the computing device determines the backbone node sequence from the input node to the M layer node of the measured network using the straightening algorithm. Then, the computing device searches the cumulative error accuracy interval from the backbone node sequence using a search algorithm (for example, a binary search algorithm). Optionally, in the case of searching multiple subnetworks, the computing device searches the cumulative error accuracy interval from multiple subnetworks in parallel.

[0139] In step S263, the computing device obtains the K layer corresponding reference input data from the reference network data. The computing device inputs the K layer corresponding reference input data to the intercepted K layer node to the M layer node. The computing device re-runs the K layer to the M layer node through the first processor to process the K layer corresponding reference input data, and obtains the re-generated output data (second output data) of the M layer subnetwork. The computing device compares the re-generated output data of the M layer node with the M layer corresponding reference output data. Through the selection of the K layer, the range of K-M is gradually narrowed, and the accurate positioning of the cumulative error is realized.

[0140] If the output data of the re-generated K layer to M layer subnetwork has accuracy abnormality, the cumulative error accuracy interval determines that the K layer to M layer subnetwork is the cumulative error interval.

[0141] Optionally, if the P layer between the K layer and the M layer in the K layer to M layer subnetwork also needs input data, the P layer corresponding reference input data is obtained from the reference network data, and the P layer corresponding reference input data is input to the P layer in the K layer to M layer subnetwork.

[0142] In step S280, the computing device obtains the (M+1) layer subnetwork corresponding reference input data from the reference network data, inputs the (M+1) layer subnetwork corresponding reference input data to the (M+1) layer subnetwork in the measured network, re-runs the (M+1) layer subnetwork to the last layer subnetwork through the first processor, obtains the output data of the last layer subnetwork, analyzes the accuracy problem of the subsequent subnet of the M layer based on the output data of the last layer subnetwork and the last layer subnetwork corresponding reference output data, and re-executes step S220.

[0143] In one possible implementation, in the appendix Figure 1 In the method shown, the computing device determines the first sub-network using a linearization algorithm. For example, in the attached... Figure 3 In the method shown, the computing device uses a linearization algorithm to search for the cumulative error accuracy range in the sub-network from layer K to layer M.

[0144] For example, in the appendix Figure 1 In the method shown, the computing device determines the backbone node sequence from the input node to the first node of the network under test using a linearization algorithm; and determines the second sub-network from the backbone node sequence; in other words, each node in the second sub-network is a backbone node in the network under test, and the first sub-network is a sequence of sub-nodes in the backbone node sequence.

[0145] In one possible implementation, the network under test includes a second node and a third node, the third node being a child node of the second node. The traffic value of the third node is determined based on the traffic value of the second node and the number of child nodes of the second node. In response to determining that the traffic value of the third node is the same as the traffic value of the input node, the third node is added to the backbone node sequence.

[0146] In one possible implementation, to determine the backbone node sequence, the computing device traverses every node in the network under test from the input node to the Mth layer node. For the currently traversed node, the computing device determines the current node's traffic value and compares it with the traffic value of the input node. If the current node's traffic value is equal to the input node's traffic value, the computing device adds the current node to the backbone node sequence. Furthermore, based on the current node's traffic value and its out-degree, the computing device proportionally allocates traffic values ​​to each of the current node's child nodes. The allocated traffic value to each child node is, for example, the ratio between the current node's traffic value and its out-degree. For example, a node's traffic value is the sum of the ratios between the traffic values ​​of each of its parent nodes and the corresponding parent node's out-degree.

[0147] Flow values ​​describe the information transmitted between nodes in a network. Flow values ​​represent the dependencies between nodes and the direction of data flow. A node's in-degree is the number of its child nodes; it can also be understood as the number of edges pointing to that node. A node's out-degree is the number of its parent nodes; it can also be understood as the number of edges from that node to other nodes. In-degree and out-degree are used to describe the connectivity and dependencies of nodes in the network.

[0148] For example, the process of determining the backbone node sequence includes the following steps A through D.

[0149] Step A, according to the given constant C, set the flow value flow = C. The flow value flow is usually the in-degree value of the input node, which is used to determine the condition of the backbone node.

[0150] Step B, initialize the flow value nodeflow[node] of each node from the input node to the Mth node to 0.0, and initialize the flow value of the input node to C.

[0151] Step C, for each node from the input node to the Mth node, calculate the flow value nodeflow[node] of the node.

[0152] For example, the out-degree value of each node is out[node], and the in-degree value of each node is in[node], the starting point is added to the BFS queue, and the breadth-first search (BFS) is started, as step four.

[0153] Step D, establish a BFS queue, which is used to save the backbone node sequence. Traverse each node from the input node to the Mth node, and for the currently traversed node, determine whether the flow value nodeflow[node] of the current node is equal to the flow value. If the flow value nodeflow[node] of the current node is equal to the flow value, the current node is determined as the backbone node, and the current node is added to the backbone node sequence. In addition, based on the flow value nodeflow[son] of the current node and the out-degree out[node] of the current node, the flow value of each child node of the current node is determined.

[0154] For example, please refer to the attached Figure 4 , the attached Figure 4 determines the specific example of the backbone node sequence using the straightening algorithm. The attached Figure 4The network 400 shown is a specific example of a network under test, and the input node (the start node, the first layer node) of the network 400 is node 410. The traffic value of the node 410 is 3. The node 410 has three child nodes, which are node 421, node 422 and node 350. The out-degree of the node 410 is 3. The ratio between the traffic value (3) of the node 410 and the out-degree (3) of the node 410 is 1. Therefore, the node 410 allocates a traffic value of 1 to the node 421, the node 422 and the node 350. The traffic values of the node 421 and the node 422 are both 1. The node 421 has two child nodes. The two child nodes of the node 421 are node 431 and node 432. The out-degree of the node 421 is 2. The ratio between the traffic value (1) of the node 421 and the out-degree (2) of the node 421 is one-half, and therefore the node 421 allocates a traffic value of one-half to the node 431 and the node 432. The traffic values of the node 431 and the node 432 are both one-half. The child node of the node 431 and the child node of the node 432 are both node 441. The out-degree of the node 431 and the out-degree of the node 432 are both 1, and therefore the node 431 and the node 432 allocate a traffic value of one-half to the node 441. The traffic value allocated to the node 441 is the sum of the traffic value (one-half) of the node 431 and the traffic value (one-half) of the node 432, and the traffic value of the node 441 is 1. The child node of the node 422 is node 442, and the ratio between the traffic value (1) of the node 422 and the out-degree (1) of the node 422 is 1, and a traffic value of 1 is allocated to the node 442. The node 441 allocates a traffic value of 1 to the node 450. The node 442 allocates a traffic value of 1 to the node 450. The traffic value of the node 450 is 1+1+1=3. The child node of the node 450 is node 460, and the node 450 allocates a traffic value of 3 to the node 460. In the network 400, the traffic values of the three nodes, the node 410, the node 450 and the node 460, are all equal to the initial traffic value (3) set, and therefore the node 410, the node 450 and the node 460 are added to the backbone node sequence, and the final backbone node sequence includes the node 410, the node 450 and the node 460

[0155] By the process of determining the backbone node sequence, the determined backbone node sequence includes a complete subnetwork, and the determined backbone node sequence is in a straight line, the backbone node sequence structure is linear without branches or redundant process nodes, so that the process nodes are not cut into the subnetwork, and thus it is not necessary to perform additional input reference data and compare the output results of the process nodes, so as to reduce the operation complexity of the input data and the comparison of the output results, and further improve the positioning accuracy error speed.

[0156] In a possible implementation, in the appended Figure 1In the method, the computing device determines the first sub-network in a recursive manner. For example, in the appended Figure 3 In the method, the computing device searches the cumulative error precision interval from the K-layer to the M-layer sub-network in a recursive manner.

[0157] For example, the computing device determines the second sub-network based on the measured network, the head node of the second sub-network is the input node of the measured network, and the tail node of the second sub-network is the first node; the computing device inputs the reference input data corresponding to the second sub-network into the second sub-network, and runs the second sub-network to obtain the output data of the second sub-network; and the computing device determines the first sub-network from the second sub-network based on the accuracy check result of the output data of the second sub-network. By determining the first sub-network from the second sub-network based on the accuracy check result of the second sub-network, the range of the cumulative error precision interval is narrowed from the second sub-network to the first sub-network, and the accuracy of the cumulative error precision interval is improved.

[0158] In a possible implementation, the computing device determines the first sub-network based on the accuracy check result of the interval from the intermediate node to the end node in the second sub-network. Specifically, the computing device determines the intermediate node of the second sub-network based on the head node of the second sub-network and the tail node of the second sub-network. The computing device intercepts the network between the intermediate node of the second sub-network and the first node of the second sub-network to obtain a fourth sub-network. The computing device inputs the reference input data corresponding to the intermediate node into the intermediate node, runs the fourth sub-network to obtain the third output data of the first node, and determines the head node of the first sub-network based on the intermediate node of the second sub-network and the tail node of the second sub-network in response to determining that the accuracy of the third output data of the first node is abnormal, or determines the tail node of the first sub-network based on the intermediate node of the second sub-network and the head node of the second sub-network in response to determining that the accuracy of the third output data of the first node is normal.

[0159] In a possible implementation, the computing device locates the first sub-network by using a binary search algorithm, which is illustrated by steps 1 to 2.

[0160] Step 1, the computing device sets the head node of the second sub-network and the tail node of the second sub-network.

[0161] The computing device sets the input node of the measured network as the head node of the second sub-network. The computing device sets the node (for example, the M-layer node) with the accuracy anomaly of the output data as the tail node of the second sub-network.

[0162] Step two, the computing device performs a binary algorithm to update the position of the intermediate node in the second sub-network by iteration until the position of the head node is before the tail node, the position of the head node is the same as the tail node or the position of the intermediate node is the same as the head node, thereby gradually narrowing the range of the second sub-network.

[0163] For example, step two includes steps (2-1) to (2-4).

[0164] Step (2-1) the computing device calculates the position of the intermediate node of the second sub-network based on the position of the head node of the second sub-network and the position of the tail node of the second sub-network.

[0165] For example, the midpoint between the position of the head node of the second sub-network and the position of the tail node of the second sub-network is determined as the position of the intermediate node of the second sub-network. For example, the position of the intermediate node is obtained by the following formula (1).

[0166] mid = (head + tail + 1) / 2; (1)

[0167] Wherein, mid represents the position of the intermediate node, head represents the position of the head node, and tail represents the position of the tail node.

[0168] Step (2-2) the computing device cuts off the first sub-network from the intermediate node to the node where the output data precision is abnormal from the second sub-network. The computing device checks whether the precision of the first sub-network is normal. For example, the computing device inputs the reference input data corresponding to the intermediate node into the first sub-network, executes the first sub-network, and obtains the output data. The computing device compares the output data with the reference output data. If the difference between the output data and the reference output data is greater than or equal to the threshold value, the computing device determines that the checking result is precision abnormal, i.e. there is an accuracy error problem. If the difference between the output data and the reference output data is less than or equal to the threshold value, the computing device determines that the checking result is precision normal.

[0169] Step (2-3) if the checking result is precision abnormal, the computing device sets the intermediate node of the second sub-network as a new head node. In other words, the head node of the first sub-network is the intermediate node of the second sub-network.

[0170] Since the checking result of the first sub-network from the intermediate node to the tail node is precision abnormal, it indicates that the cumulative error precision interval is included from the intermediate node to the tail node. By resetting the head node, it is equivalent to moving the head node to the position of the intermediate node, and re-dividing the first sub-network from the new intermediate node to the tail node, thereby narrowing the range of the cumulative error precision interval searched by half, excluding the range from the original head node to the intermediate node, and improving the accuracy of the cumulative error precision interval.

[0171] If the verification result is normal accuracy, the computing device sets the previous node of the intermediate node of the second sub-network as the new tail node. In other words, the tail node of the first sub-network is the previous node of the intermediate node of the second sub-network.

[0172] Since the verification result of the first sub-network is normal accuracy, it indicates that the interval of accumulated error accuracy does not include the intermediate node to the tail node, and thus the interval of accumulated error accuracy can be excluded from the range of the intermediate node to the tail node. Since it has been verified that the interval of accumulated error accuracy includes the head node to the tail node, it can be determined that the interval of accumulated error accuracy is located from the head node to the previous node of the intermediate node. By resetting the tail node, the tail node is moved to the previous node of the intermediate node, and the first sub-network from the head node to the new tail node is re-divided, so as to reduce the range of the searched interval of accumulated error accuracy by half, and improve the accuracy of the interval of accumulated error accuracy.

[0173] For example, please refer to the accompanying drawings Figure 5 , the accompanying drawings Figure 5 show a specific example of searching for the interval of accumulated error accuracy by using binary search. The accompanying drawings Figure 5 show a three-time binary search process. In the first binary search process, as shown in (a) of the accompanying drawings Figure 5 , in the process of performing step (2-2), the computing device intercepts the sub-network from the intermediate node 512 to the tail node 513 from the measured network to obtain the sub-network 510; the computing device performs accuracy verification on the sub-network 510; the computing device determines that the accuracy of the sub-network 510 is normal, and then performs step (2-4) to set the previous node of the intermediate node 512 as the new tail node, so as to determine the tail node 523. In the second binary search process, as shown in (b) of the accompanying drawings Figure 5 , in the process of performing step (2-1), the computing device determines the intermediate node 522 based on the head node 511 and the tail node 523. In the process of performing step (2-2), the computing device intercepts the network from the intermediate node 522 to the terminal node from the measured network to obtain the sub-network 520. In the process of performing step (2-2), the computing device performs accuracy verification on the sub-network 520, and the computing device determines that the accuracy of the sub-network 520 is normal, and then performs step (2-4) to set the previous node of the intermediate node 522 as the new tail node, so as to determine the tail node 533. In the third binary search process, as shown in (c) of the accompanying drawings Figure 5As shown in (c) of FIG. 5, in the process of performing step (2-1), the computing device determines the intermediate node 532 based on the head node 511 and the tail node 533. In the process of performing step (2-2), the computing device intercepts the network from the intermediate node 532 to the end node from the measured network to obtain the sub-network 530. In the process of performing step (2-2), the computing device performs precision checking on the sub-network 530. If the computing device determines that the precision of the sub-network 530 is abnormal, the computing device performs step (2-3) to set the intermediate node 532 as a new head node, thereby determining the head node 541. The computing device outputs the identification of the head node 541 and the identification of the end node, indicating that the accumulated precision error interval is from the head node 541 to the end node.

[0174] Exemplarily, the computing device executes the following program code to determine the accumulated precision error interval.

[0175] while head<tail:

[0176] mid=(head+tail+1) / 2

[0177] #check the connected graph from the mid node to the end node

[0178] #whether the precision is normal

[0179] res=check(mid,endnode)

[0180] #precision abnormal

[0181] if res:

[0182] head=mid

[0183] #precision error interval

[0184] else:

[0185] tail=mid–1

[0186] #return the start node of the precision error interval and the end node of the precision error interval

[0187] return head,endnode

[0188] In another possible implementation, the computing device determines the accumulated error precision interval by using the ternary method. For example, as shown in FIG. 6, the computing device determines the accumulated error precision interval by using the ternary method. Figure 6As shown, the computing device determines the quantile α and the quantile (1-α) from the current search interval (e.g., the second sub-network) according to a proportion constant α. The proportion constant α is typically one-third. The quantile α refers to a point obtained by dividing the current search interval according to the proportion constant α. The quantile α is calculated, for example, by proportionally weighting the start point and the end point of the current search interval according to α. That is, the quantile α divides the entire search interval into an α portion and a (1-α) portion, and the α quantile is located at the position occupied by the α portion of the entire search interval. The (1-α) quantile is located at the position occupied by the (1-α) portion of the entire search interval.

[0189] The quantile α is determined by the following equation (2).

[0190] α-divide = (1-α) * head + α * tail; (2)

[0191] The quantile (1-α) is determined by the following equation (3).

[0192] (1-α)-divide = α * head + (1-α) * tail; (3)

[0193] The check(startnode, endnode) function is executed to perform precision checking on the interval [startnode, endnode]. If the return result of the check function is true, it indicates that the interval [startnode, endnode] is abnormal in precision; if the return result of the check function is false, it indicates that the interval [startnode, endnode] is normal in precision. In addition, the next search interval is determined according to the interval precision checking results check(α, endnode) and check(1-α, endnode). Specifically, if check(α, endnode) returns true and check(1-α, endnode) returns true, that is, both the interval (α, endnode) and the interval (1-α, endnode) are abnormal in precision, the next search interval is [(1-α)-divide, tail].

[0194] In another possible implementation, the first child network is obtained by intercepting the parent node of the first node and the first node in the network under test.

[0195] In an example scenario, the network under test is an image recognition model. The computing device inputs an original image into the image recognition model, and identifies the original image through the image recognition model to obtain output data of each node in the image recognition model. For example, the image recognition model includes an input layer node, a convolution layer node, a pooling layer node, and an output layer node. The output data of the input layer node represents pixels in the original image or features of the original image. The convolution layer node performs a convolution operation on the original image through a convolution kernel to output local features in the original image. The pooling layer node outputs the most significant features in a feature map. The output data of the output layer node is a probability value, which represents a possibility that the image belongs to a type.

[0196] In an exemplary scenario, the network under test is a text recognition model. The computing device inputs raw text into the text recognition model, which then recognizes the raw text to obtain the output data for each node. Input layer nodes convert the raw text into character, word, or encoded vector representations. The output data of each input layer node is, for example, a character, word, or feature from the raw text. Embedding layer nodes map discrete representations of characters, words, or features to a continuous low-dimensional vector space, outputting word embedding vectors or character embedding vectors. Convolutional layers perform convolution operations on the input sequence using sliding convolution kernels to extract local combined features. Each node corresponds to a specific feature extracted by a convolution kernel at a certain position in the input sequence. Recurrent layer nodes extract the temporal relationships of the input sequence using Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU), modeling the contextual information based on these temporal relationships. Attention layer nodes assign weights to different parts of the input sequence and combine the attention-weighted representation with the contextual information. The output data of each attention layer node is, for example, an attention-weighted representation or a context vector.

[0197] Appendix Figure 7 This is a flowchart of a precision error positioning method provided in an embodiment of this application. Figure 7 The method shown is a system-level precision error positioning method, with appendix. Figure 7 The method shown supports parallel monitoring of the accuracy of each layer of operators in an AI model during training or inference operations based on the AI ​​model. (See attached...) Figure 7 The method shown allows computing devices to analyze the accuracy of operators at each layer of an AI model in real time while performing training or inference operations based on the AI ​​model. (Appendix) Figure 7 The method shown can locate the accuracy error of both operators and subnetworks. (Appendix) Figure 7 It is attached Figure 3 The method shown is applied to specific AI training and inference business scenarios with concrete examples. (Attached) Figure 7 The network under test in the method shown is, for example, an AI model. Figure 7 In the method shown, the output data of each layer of the network is attached. Figure 3 A specific example of the output data of the subnetwork in the method shown is attached. Figure 7 The operators in the illustrated method are concrete examples of nodes. (Appendix) Figure 7 The method shown includes the following steps.

[0198] Step S601: The computing device constructs a DAG graph of the network under test to obtain the DAG of the network under test.

[0199] A Directed Acyclic Graph (DAG) indicates the topology of the network under test. The DAG includes information about the nodes at each layer of the network. By constructing the DAG, subgraphs can be extracted based on it.

[0200] In step S602, the computing device runs the measured network to obtain output data of each layer in the measured network. The computing device saves the output data of each layer in the measured network in the memory. The computing device can save the output data of each layer in the measured network in the hard disk or in the memory.

[0201] In a possible implementation, the computing device detects the life cycle of the output data of each layer saved in the memory. In response to detecting that the life cycle of the output data of each layer ends, the computing device deletes the saved output data of each layer from the memory, thereby saving the occupied storage space.

[0202] In step S603, the computing device performs accuracy comparison between the output data of each operator in the measured network data and the output data of each operator in the benchmark data based on the saved measured network data and the benchmark data.

[0203] In step S604, if the first operator with accuracy problem in the measured network data is located based on the accuracy comparison result, the computing device extracts the first operator with accuracy problem from the measured network data, inputs the benchmark input data to the operator, executes the operator to obtain the output data of the operator. The computing device compares the output data of the operator with the benchmark output data corresponding to the operator. The computing device determines whether the operator has accuracy problem based on the comparison result.

[0204] In step S605, if the extracted single operator has no accuracy problem, the computing device determines that the accuracy error of the operator in the layer is caused by the accumulated error.

[0205] In step S606, the computing device splits the measured network into one or more subgraphs based on the DAG.

[0206] In step S607, the computing device extracts the backbone node sequence from the one or more subgraphs of the measured network by using the straight-line algorithm.

[0207] In step S608, the computing device inputs the benchmark data to the backbone node sequence, executes the backbone node sequence to obtain the output data of the backbone node sequence.

[0208] In step S609, the computing device compares the output data of the backbone node sequence with the benchmark output data of the tail node corresponding to the backbone node sequence.

[0209] In step S610, the computing device locates the accumulated accuracy error interval based on the comparison result.

[0210] Optionally, in the case of extracting a plurality of trunk node sequences, the computing device performs the precision error positioning process of the plurality of trunk node sequences in parallel. For example, the computing device inputs the reference input data to the head node of each trunk node sequence in parallel, the computing device executes each trunk node sequence in parallel to obtain the output data of the tail node of each trunk node sequence; the computing device compares the output data of the tail node of each trunk node sequence with the reference output data corresponding to the tail node in parallel, thereby judging in parallel whether each trunk node sequence includes the cumulative precision error interval.

[0211] Optionally, the computing device further determines a sub-node sequence from the trunk node sequence in a recursive manner, judges whether the sub-node sequence includes the cumulative precision error interval based on the output data of the sub-node sequence, thereby narrowing the range of the determined cumulative precision error interval and improving the accuracy of the cumulative precision error interval.

[0212] Step S611, the computing device uses a dichotomy algorithm to extract a sub-node sequence from the trunk node sequence.

[0213] Step S612, the computing device inputs the reference data to the sub-node sequence, executes the sub-node sequence to obtain the output data of the sub-node sequence.

[0214] Step S613, the computing device compares the output data of the sub-node sequence with the reference output data of the sub-node sequence.

[0215] Step S614, the computing device judges whether the sub-node sequence includes the cumulative error precision interval based on the comparison result, thereby realizing the positioning of the cumulative error precision interval.

[0216] Step S615, if the output data of the single operator has a precision problem, the computing device extracts the reference input data of the operator from the reference network data, inputs the reference input data to the operator, and executes the operator to obtain the output data of the operator.

[0217] Step S616, the computing device compares the output data of the operator with the reference output data corresponding to the operator to obtain a comparison result.

[0218] Step S617, the computing device determines that the precision problem of the output data of the operator is caused by the operator itself based on the comparison result.

[0219] Step S618, the computing device inputs the reference data to the subsequent cascaded sub-network of the abnormal operator.

[0220] Step S619, the computing device runs the subsequent cascaded sub-network based on the reference data, thereby analyzing the precision of the subsequent cascaded sub-network.

[0221] In step S621, if the single operator itself has an accuracy problem, the computing device performs DAG mapping on the matrix operation instruction, the vector operation instruction and the scalar operation instruction contained in the operator.

[0222] In step S622, the computing device inputs the reference input tensor to the operator, executes the operator to obtain an output tensor of the operator, and compares the output tensor of the operator with the output tensor of the CPU.

[0223] In step S623, the computing device uses a dichotomy or other searching algorithm to intercept part of the instructions from the DAG of the operator, executes the intercepted instructions by the first processor based on the reference input tensor to obtain an output tensor of the instructions, or intercepts the upper interface encapsulated by the instructions from the DAG of the operator, inputs the reference input tensor to the interface, calls the interface by the first processor to obtain an output tensor of the interface.

[0224] In step S624, the computing device compares the output tensor of the instructions executed by the first processor with the output tensor of the instructions executed by the second processor (such as the CPU) to obtain a comparison result. Alternatively, the computing device compares the output tensor of the interface called by the first processor with the output tensor of the interface called by the second processor (such as the CPU) to obtain a comparison result.

[0225] In step S625, if the comparison result of the output tensor of the instructions indicates an accuracy abnormality, the computing device determines that the cumulative error accuracy interval is included in the intercepted instructions, thereby identifying which instructions cause the accuracy error of the operator. If the comparison result of the output tensor of the interface indicates an accuracy abnormality, the computing device determines that the cumulative error accuracy interval is included in the intercepted interface, thereby identifying which interface causes the accuracy error of the operator.

[0226] In step S618, the computing device inputs the reference input data corresponding to the subsequent cascaded subnetwork to the subsequent cascaded subnetwork of the operator whose output data has an accuracy abnormality.

[0227] In step S619, the computing device runs the subsequent cascaded subnetwork to determine the accuracy problem of the subsequent cascaded subnetwork.

[0228] The method provided in this embodiment compares the actual output data at the levels of the operator, the instruction, the interface and the network with the corresponding reference output data, realizes full analysis of the accuracy error operator of the whole network, and can directly locate the layer of the operator in the whole network that has an accuracy problem, or the accuracy problem caused by the cumulative error of several layers of operators, or even the operator accuracy problem caused by the cumulative error of several instructions of the operator that has an accuracy problem.

[0229] FIG. 1 is a flowchart of an accuracy error positioning method provided by an embodiment of the present application. Figure 8 is a flowchart of an accuracy error positioning method provided by an embodiment of the present application.Figure 8 The method can be executed in parallel with training service or inference service. Figure 8 The method comprises steps S710 to S780.

[0230] At step S710, the computing device acquires a pre-trained framework model. The pre-trained framework model is, for example, a pytorch model, a tensorflow model, a paddlepaddle model, an ONNX model, or a mindspore framework model.

[0231] At step S720, the computing device parses the pre-trained framework model to generate a DAG graph of the pre-trained framework model, so as to uniformly use a graph to express various types of pre-trained framework models. The DAG graph of the pre-trained framework model comprises an upstream and downstream relationship of each layer operator in the network.

[0232] Step S720 comprises steps S721 to S723.

[0233] At step S721, the computing device parses the DAG of the pre-trained framework model to obtain a parsing result.

[0234] At step S722, the computing device generates a native intermediate representation (IR) of the pre-trained framework model based on the parsing result.

[0235] At step S723, the computing device generates a DAG of the pre-trained framework model based on the IR.

[0236] At step S730, the computing device extracts a subnetwork from the DAG.

[0237] At step S740, the computing device extracts reference input data of the subnetwork.

[0238] At step S750, the computing device performs precision error positioning for multiple subnetworks in parallel.

[0239] At step S760, the computing device performs cumulative error positioning on an upper subnetwork.

[0240] At step S770, the computing device performs precision error positioning on a subsequent subnetwork of an operator with abnormal precision.

[0241] At step S780, the computing device executes a training service or an inference service.

[0242] Step S780 can be executed in parallel with each of steps S710 to S770.

[0243] The embodiment provides a method suitable for reasoning and training, which can bound and locate precision problems caused by accumulated errors of operators. In addition, the method can effectively locate precision problems of subsequent operators of an abnormal-precision operator.

[0244] In addition, a model precision error all-around analysis method which can be automatically implemented based on a network topology is provided. Thus, from original manual serial positioning, system-level parallel operator problem positioning is realized.

[0245] In addition, an end-to-end precision real-time analysis system method is provided, which can directly locate precision problems of operator instructions from precision problems of a training model and directly locate precision problems of operator instructions from precision problems of a reasoning model. The method can realize real-time analysis and positioning of operator precision problems in training and reasoning processes.

[0246] In addition, a general precision error positioning process suitable for reasoning and training is provided, which can distinguish error types and error positions, adopts a linearization algorithm to simplify a network topology, and uses a search algorithm to improve search efficiency.

[0247] The embodiment of the application further provides a precision error positioning device, as shown in the accompanying Figure 9 The precision error positioning device 900 comprises:

[0248] A determination module 910 is configured to determine that precision of first output data of a first node in a measured network is abnormal.

[0249] A running module 920 is configured to input reference input data corresponding to the first node into the first node and run the first node to obtain second output data of the first node.

[0250] The determination module 910 is further configured to determine that precision of the second output data of the first node is normal.

[0251] A cutting module 930 is configured to cut a first sub-network from the measured network.

[0252] The running module 920 is further configured to input reference input data corresponding to the first sub-network into the first sub-network and run the first sub-network to obtain output data of the first sub-network, the first sub-network being an upper network of the first node in the measured network.

[0253] The determination module 910 is further configured to determine that the first sub-network comprises an accumulated error precision interval in response to determining that precision of the output data of the first sub-network is abnormal.

[0254] In a possible implementation, the intercepting module 930 is configured to determine a second sub-network based on the measured network, a head node of the second sub-network being the input node of the measured network, and a tail node of the second sub-network being the first node; input reference input data corresponding to the second sub-network into the second sub-network, and run the second sub-network to obtain output data of the second sub-network; and determine the first sub-network from the second sub-network based on an accuracy checking result of the output data of the second sub-network.

[0255] In a possible implementation, the second sub-network includes an intermediate node between the head node and the tail node, and the running module 920 is configured to input reference input data corresponding to the intermediate node into the intermediate node, run a sub-network from the intermediate node to the tail node in the second sub-network, and obtain the output data of the second sub-network.

[0256] The determining module 910 is configured to determine the head node of the first sub-network based on the intermediate node of the second sub-network and determine the tail node of the first sub-network based on the tail node of the second sub-network in response to determining that the accuracy of the output data of the second sub-network is abnormal, or determine the tail node of the first sub-network based on the intermediate node of the second sub-network and determine the head node of the first sub-network based on the head node of the second sub-network in response to determining that the accuracy of the output data of the second sub-network is normal.

[0257] In a possible implementation, the determining module 910 is configured to determine a sequence of backbone nodes from the input node to the first node in the measured network, and determine the second sub-network from the sequence of backbone nodes.

[0258] In a possible implementation, the determining module 910 is configured to intercept a parent node of the first node and the first node in the measured network to obtain the first sub-network.

[0259] In a possible implementation, the intercepting module 930 is further configured to intercept a third sub-network from the measured network, input reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network to obtain output data of the third sub-network, and the third sub-network is a lower network of the first node in the measured network.

[0260] The determining module 910 is further configured to determine a node with accuracy abnormality in the third sub-network in response to determining that the accuracy of the output data of the third sub-network is abnormal.

[0261] In a possible implementation, the determining module 910 is further configured to determine that the first node has accuracy abnormality in response to determining that the accuracy of the second output data of the first node is abnormal.

[0262] In a possible implementation, the network under test includes a second node and a third node, the third node being a child node of the second node, the determining module 910 is configured to determine a traffic value of the third node based on a traffic value of the second node and a number of child nodes of the second node, and in response to determining that the traffic value of the third node is the same as the traffic value of the input node, add the third node to the sequence of backbone nodes.

[0263] The determining module 910, the running module 920, and the intercepting module 930 can be implemented by software or by hardware. For example, the implementation of the determining module 910 is described below. The implementation of the running module 920 and the intercepting module 930 can be similar to the implementation of the determining module 910.

[0264] As an example of a software functional unit, the determining module 910 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the determining module 910 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each AZ including one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0265] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to be set through a communication gateway in each VPC to realize the interconnection between VPCs.

[0266] As an example of a hardware functional unit of a module, the determining module 910 can include at least one computing device, such as a server or the like. Alternatively, the determining module 910 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0267] The plurality of computing devices included in the determining module 910 can be distributed in the same region or in different regions. The plurality of computing devices included in the determining module 910 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the determining module 910 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0268] It should be noted that in other embodiments, the determining module 910 can be configured to perform any step of the precision error positioning method, the running module 920 can be configured to perform any step of the precision error positioning method, and the intercepting module 930 can be configured to perform any step of the precision error positioning method. The steps implemented by the determining module 910, the running module 920, and the intercepting module 930 can be specified as needed, and the overall function of the precision error positioning device can be achieved by implementing different steps of the precision error positioning method by the determining module 910, the running module 920, and the intercepting module 930, respectively.

[0269] The present application also provides a computing device 100. As shown in Figure 10 The computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0270] The bus 102 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one line is used in the figure to represent the bus, but this does not mean there is only one bus or only one type of bus. The bus 104 can include paths for communicating information among the various components of the computing device 100 (e.g., the memory 106, the processor 104, the communication interface 108).

[0271] The processor 104 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0272] The memory 106 can include a volatile memory (e.g., random access memory (RAM)), and the processor 104 can further include a non-volatile memory (e.g., read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0273] The memory 106 stores executable program code, and the processor 104 executes the executable program code to respectively implement the functions of the foregoing determination module 910, the running module 920, and the intercepting module 930, so as to implement the precision error positioning method. That is, the memory 106 has instructions for executing the precision error positioning method.

[0274] The communication interface 103 uses a transceiving module such as, but not limited to, a network interface card or a transceiver to implement the communication between the computing device 100 and other devices or communication networks.

[0275] The embodiments of the present application also provide a computing device cluster. The computing device cluster comprises at least one computing device. The computing device can be a server, for example, a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smart phone.

[0276] As shown in Figure 11 , the computing device cluster comprises at least one computing device 100. The same instructions for performing the precision error positioning method can be stored in the memory 106 of one or more computing devices 100 in the computing device cluster.

[0277] In some possible implementations, partial instructions for performing the precision error positioning method can also be respectively stored in the memory 106 of one or more computing devices 100 in the computing device cluster. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for performing the precision error positioning method.

[0278] It should be noted that the memories 106 in different computing devices 100 in the computing device cluster can store different instructions, respectively used to perform partial functions of the precision error positioning apparatus. That is, the instructions stored in the memories 106 in different computing devices 100 can implement the functions of one or more of the determination module 910, the running module 920, and the intercepting module 930.

[0279] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. Figure 12 A possible implementation is shown. As shown in Figure 12 , two computing devices 100A and 100B are connected through a network. Specifically, the communication interfaces in the respective computing devices are connected to the network. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for performing the functions of the determination module 910. Meanwhile, the memory 106 in the computing device 100B stores instructions for performing the functions of the running module 920 and the intercepting module 930.

[0280] Figure 12 The connection mode between the computing device cluster shown in

[0281] It should be understood that Figure 12The functions of the computing device 100A shown in FIG. 1 can also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be completed by multiple computing devices 100.

[0282] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection relationship between the computing devices in the computing device cluster 100A shown in FIG. 1. Figure 11 and Figure 12 The connection manner of the computing device cluster is similar to the connection manner of the computing device cluster 100A shown in FIG. 1. The difference is that the same instructions for performing the precision error positioning method can be stored in the memory 106 of one or more computing devices 100 in the computing device cluster.

[0283] In some possible implementation manners, the memory 106 of one or more computing devices 100 in the computing device cluster can also respectively store partial instructions for performing the precision error positioning method. In other words, the combination of the one or more computing devices 100 can collectively execute the instructions for performing the precision error positioning method.

[0284] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the precision error positioning method.

[0285] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the precision error positioning method.

[0286] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts of each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments.

[0287] A refers to B, which means that A is the same as B or a simple transformation of B.

[0288] The terms "first" and "second" and the like in the specification and claims of the embodiments of the present application are used to distinguish different objects, and are not used to describe a specific order of the objects, nor can be understood as indicating or implying relative importance. For example, the first sub-network and the second sub-network are used to distinguish different sub-networks, and are not used to describe a specific order of the sub-networks, nor can be understood as the first sub-network being more important than the second sub-network.

[0289] In this application, unless otherwise stated, the meaning of "at least one" is one or more, and the meaning of "multiple" is two or more. For example, multiple sub-networks refer to two or more sub-networks.

[0290] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the whole or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)), etc.

[0291] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A precision error localization method, characterized in that, The method comprises: in response to determining that the accuracy of the first output data of the first node in the measured network is abnormal, inputting the reference input data corresponding to the first node into the first node, running the first node to obtain second output data of the first node; in response to determining that the accuracy of the second output data of the first node is normal, intercepting a first sub-network from the measured network, inputting the reference input data corresponding to the first sub-network into the first sub-network, running the first sub-network to obtain output data of the first sub-network, the first sub-network being an upper network of the first node in the measured network; in response to determining that the accuracy of the output data of the first sub-network is abnormal, determining that the first sub-network comprises a cumulative error accuracy interval.

2. The method of claim 1, wherein, The first sub-network is intercepted from the measured network, comprising: determining a second sub-network based on the measured network, a head node of the second sub-network being an input node of the measured network, and a tail node of the second sub-network being the first node; inputting reference input data corresponding to the second sub-network into the second sub-network, running the second sub-network to obtain output data of the second sub-network; determining the first sub-network from the second sub-network based on the accuracy checking result of the output data of the second sub-network.

3. The method of claim 2, wherein, The second sub-network comprises intermediate nodes between the head node and the tail node, and the inputting of the reference input data corresponding to the second sub-network into the second sub-network, the running of the second sub-network to obtain the output data of the second sub-network, comprises: inputting reference input data corresponding to the intermediate nodes into the intermediate nodes, running a sub-network from the intermediate nodes to the tail nodes in the second sub-network to obtain the output data of the second sub-network; The determination of the first sub-network from the second sub-network based on the accuracy checking result of the output data of the second sub-network, comprises: in response to determining that the accuracy of the output data of the second sub-network is abnormal, determining the head node of the first sub-network based on the intermediate nodes of the second sub-network, and determining the tail node of the first sub-network based on the tail node of the second sub-network; or, in response to determining that the accuracy of the output data of the second sub-network is normal, determining the tail node of the first sub-network based on the tail node of the second sub-network, and determining the head node of the first sub-network based on the head node of the second sub-network.

4. The method of claim 2, wherein, The determination of the second sub-network based on the measured network, comprises: determining a main node sequence from the input node of the measured network to the first node; determining the second sub-network from the main node sequence.

5. The method of claim 1, wherein, The first sub-network is intercepted from the measured network, comprising: intercepting the first node and a parent node of the first node in the measured network to obtain the first sub-network.

6. The method of claim 1, wherein, The method further comprises: intercepting a third sub-network from the measured network, inputting reference input data corresponding to the third sub-network into the third sub-network, running the third sub-network, and obtaining output data of the third sub-network, the third sub-network being a lower-layer network of the first node in the measured network; in response to determining that the output data of the third sub-network is abnormal in accuracy, determining that the third sub-network has a node abnormal in accuracy.

7. The method of claim 1, wherein, The method further comprises: in response to determining that the second output data of the first node is abnormal in accuracy, determining that the first node has a node abnormal in accuracy.

8. The method according to any one of claims 1 to 7, characterized in that, The first node is an Mth-layer network in the measured network, the first sub-network is a Kth-layer network to the Mth-layer network in the measured network, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

9. The method according to any one of claims 1 to 7, characterized in that, The measured network is a computing device cluster, the computing device cluster is used to execute a task sequence, the first node is a computing device or an AI accelerator card executing an Mth task in the task sequence, the first sub-network is a computing device or an AI accelerator card executing a Kth task to the Mth task in the task sequence, M is a positive integer greater than 1, and K is a positive integer less than or equal to M.

10. The method of claim 4, wherein, The measured network comprises a second node and a third node, the third node being a child node of the second node, and the determination of a backbone node sequence from an input node of the measured network to the first node comprises: determining a traffic value of the third node based on a traffic value of the second node and a number of child nodes of the second node; in response to determining that the traffic value of the third node is the same as the traffic value of the input node, adding the third node to the backbone node sequence.

11. An accuracy error locating device, characterized by, The device comprises: a determination module configured to determine whether first output data of a first node in a measured network is abnormal in accuracy; a running module configured to input reference input data corresponding to the first node into the first node, run the first node, and obtain second output data of the first node; the determination module is further configured to determine that the second output data of the first node is normal in accuracy; an intercepting module configured to intercept a first sub-network from the measured network; the running module is further configured to input reference input data corresponding to the first sub-network into the first sub-network, run the first sub-network, and obtain output data of the first sub-network, the first sub-network being an upper-layer network of the first node in the measured network; the determination module is further configured to, in response to determining that the output data of the first sub-network is abnormal in accuracy, determine that the first sub-network comprises a cumulative error accuracy interval.

12. The apparatus of claim 11, wherein, The intercepting module is configured to determine a second sub-network based on the measured network, a head node of the second sub-network being an input node of the measured network, and a tail node of the second sub-network being the first node; inputting reference input data corresponding to the second sub-network into the second sub-network, running the second sub-network, and obtaining output data of the second sub-network; and determine the first sub-network from the second sub-network based on a precision check result of output data of the second sub-network.

13. The apparatus of claim 12, wherein, The second sub-network includes an intermediate node between the head node and the tail node, and the running module is configured to input reference input data corresponding to the intermediate node into the intermediate node, run a sub-network from the intermediate node to the tail node in the second sub-network, and obtain output data of the second sub-network. The determining module is configured to determine the head node of the first sub-network based on the intermediate node of the second sub-network and determine the tail node of the first sub-network based on the tail node of the second sub-network in response to determining that the precision of the output data of the second sub-network is abnormal. Alternatively, the determining module is configured to determine the tail node of the first sub-network based on the intermediate node of the second sub-network and determine the head node of the first sub-network based on the head node of the second sub-network in response to determining that the precision of the output data of the second sub-network is normal.

14. The apparatus of claim 12, wherein, The determining module is configured to determine a backbone node sequence from an input node of the measured network to the first node.

15. The apparatus of claim 11, wherein, The determining module is configured to intercept a parent node of the first node and the first node in the measured network to obtain the first sub-network.

16. The apparatus of claim 11, wherein, The intercepting module is further configured to intercept a third sub-network in the measured network, input reference input data corresponding to the third sub-network into the third sub-network, run the third sub-network, and obtain output data of the third sub-network, the third sub-network being a lower network of the first node in the measured network. The determining module is further configured to determine a node with precision abnormality in the third sub-network in response to determining that the precision of the output data of the third sub-network is abnormal.

17. The apparatus of claim 11, wherein, The determining module is further configured to determine that the first node has precision abnormality in response to determining that the precision of the second output data of the first node is abnormal.

18. The apparatus of claim 14, wherein, The measured network includes a second node and a third node, the third node being a child node of the second node, and the determining module is configured to determine a traffic value of the third node based on a traffic value of the second node and a number of child nodes of the second node. In response to determining that the traffic value of the third node is the same as the traffic value of the input node, the third node is added to the backbone node sequence.

19. A cluster of computing devices, characterized in that, The computing device cluster is a measured network, and the computing device cluster includes at least one computing device, each computing device being a node in the measured network, and each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method of any one of claims 1 to 10.

20. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1 to 10.

Images