Neural network model migration operator test method and device and computer equipment

Abstract

The invention provides a neural network model migration operator test method and apparatus, and a computer device. The method comprises the steps of obtaining a first evaluation data set including multiple pieces of first operator input data; determining an output error threshold according to the plurality of first operator input data and the first computational graph; the first model is a neural network model under an original framework; determining a plurality of test thresholds according to the output error threshold; respectively inputting the same group of operator input data into the first calculation graph and the second calculation graph to obtain a first operator output result and a second operator output result; the second model refers to a neural network model under the current framework; calculating an error value between the output result of the first operator and the output result of the second operator; and determining operator migration information of the second model according to the error value and a plurality of test thresholds so as to optimize the second model. According to the embodiment of the invention, an error source in a neural network model migration process can be effectively positioned.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, and computer equipment for testing neural network model transfer operators. Background Technology

[0002] Framework precision alignment refers to reconstructing a model originally implemented based on the PyTorch framework by refactoring its training or inference program to reimplement it using operators and logic from other deep learning frameworks (such as PaddlePaddle), thereby forming a training or inference program for the corresponding framework.

[0003] After completing the framework conversion, it is necessary to compare and verify the accuracy of the calculation results of the obtained model with those of the original framework. Because different deep learning frameworks differ in operator implementation methods and underlying computational mechanisms, certain output errors may occur under the same input conditions. When the error is large, it is necessary to further determine whether the error is caused by differences in operator implementation methods or by defects in the operator's implementation itself.

[0004] Existing methods typically rely on manually setting fixed thresholds for judgment, or on manually constructing test cases one by one to analyze the causes of errors, which is inefficient and the results depend on human experience. Summary of the Invention

[0005] In view of this, this application proposes a method, apparatus and computer equipment for testing neural network model transfer operators, in order to solve the problem that the source of error cannot be determined in the existing neural network model transfer process.

[0006] The first aspect of this application proposes a method for testing neural network model transfer operators, the method comprising: Obtain a first evaluation dataset containing multiple first operator input data; the first evaluation dataset refers to a dataset used to evaluate model performance in real application scenarios; The output error threshold is determined based on the input data of the plurality of first operators and the first computation graph; the first computation graph represents the dependency relationship of the plurality of first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; the first model refers to the neural network model under the original framework; Multiple test thresholds are determined based on the output error threshold; The same set of operator input data is input into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result; the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the neural network model under the current framework; Calculate the error value between the output of the first operator and the output of the second operator; The operator migration information of the second model is determined based on the error value and the multiple test thresholds in order to optimize the second model.

[0007] This application embodiment obtains the first operator output result and the second operator output result by inputting the same set of operator input data into the first computation graph and the second computation graph respectively; and determines the operator transfer information of the second model by the error value between the first operator output result and the second operator output result and multiple test thresholds, so as to optimize the second model, effectively locate the error source in the neural network model transfer process, thereby improving the transfer efficiency and transfer accuracy of the network model.

[0008] In this embodiment of the application, before obtaining the first evaluation dataset containing multiple first operator input data, the method further includes: Obtain the second evaluation dataset; the second evaluation dataset includes multiple input data for the second operator; For any input data of the second operator, calculate the statistical characteristics of the input data of the second operator; The first operator input data is generated based on the statistical characteristics and the Gaussian distributed random noise. The first evaluation dataset is generated based on the input data of multiple first operators.

[0009] The embodiments of this application automatically generate a large number of new input samples by utilizing the statistical characteristics of the second operator input data and the random noise in the Gaussian distribution. This can form a more comprehensive and diverse operator input test set, significantly improving the coverage and representativeness of the operator test.

[0010] In this embodiment of the application, determining the output error threshold based on the plurality of first operator input data and the first computation graph includes: The input data of the plurality of first operators are input into the first computation graph to obtain the output result of each operator in the first computation graph; Random error terms are gradually added to the output to generate a perturbed output. The perturbated output result replaces the output result of the corresponding operator in the first computation graph, and the model output result is obtained based on the replaced first computation graph; The evaluation index value of the first model is determined based on the output result of the model. Adjust the magnitude of the random error term, repeatedly input the input data of the multiple first operators into the first computation graph, and obtain the output result of each operator in the first computation graph; gradually add random error terms to the output results to generate perturbed output results, until the evaluation index value is less than the preset index decrease threshold, and then determine the magnitude of the current random error term as the output error threshold.

[0011] This application's embodiments automatically determine the upper limit of the error, i.e., the output error threshold, by applying random perturbation errors to the operator output results under the original framework and determining the threshold point at which the error changes significantly affect the overall evaluation index. This upper limit of the error allows for the automatic evaluation of the accuracy differences in the operator output under the new framework and determines whether the error belongs to normal numerical deviations or potential operator implementation defects.

[0012] In this embodiment of the application, a plurality of first operators in the first computation graph correspond one-to-one with a plurality of second operators in the second computation graph; the first operator outputs multiple results, the second operator outputs multiple results, and the error values ​​are multiple, each error value referring to the numerical error between the corresponding first operator output result and the corresponding second operator output result; Determining the operator transfer information of the second model based on the error value and the plurality of test thresholds includes: For any one of the plurality of test thresholds, multiple numerical errors are compared with the test threshold to determine the excess ratio under the test threshold; the excess ratio is determined based on the total number of operators and the target number of operators; the target operator refers to the operator whose numerical error is greater than the test threshold. If the excess ratio is greater than a preset ratio threshold, then it is determined that the second model under the current framework has an operator precision deviation problem; If the multiple out-of-limit ratios corresponding to the multiple test thresholds are all less than or equal to the preset ratio threshold, then it is determined that the second model under the current framework does not have an operator precision deviation problem.

[0013] In this embodiment of the application, after determining that the second model under the current framework does not have an operator precision deviation problem, the method further includes: For any given test threshold, calculate the error distribution of all second operators at that test threshold; the error distribution includes the mean, variance, and out-of-limit percentage of the numerical error for each second operator; A target heatmap is generated based on multiple error distributions corresponding to multiple test thresholds; the target heatmap is used to characterize the error distribution of all second operators; The defect operator is located using the target heatmap.

[0014] In this embodiment of the application, when it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes: For any second operator output, if the numerical error between the second operator output and the corresponding first operator output is greater than the maximum test threshold, and the element size of the first operator input data corresponding to the second operator output is greater than the preset size threshold, then it is determined that the operator accuracy deviation problem is caused by the implementation defect of the second operator corresponding to the second operator output in the second computation graph.

[0015] In this embodiment of the application, each first operator input data includes multiple input elements; when it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes: For any input element in any first operator input data, calculate the position coordinates of the input element in the total element distribution of the plurality of input elements; The number of input elements in the distribution boundary region is determined based on the position coordinates of each input element, and the proportion of input elements in the distribution boundary region is calculated based on the number of input elements and the total number of input elements in the first operator input data. If the proportion of the input elements is greater than a preset proportion threshold, then the second operator in the second computation graph corresponding to the input data of the first operator is initially determined as the problem operator; Multiple sampling and clustering are performed on all input elements in the input data of the first operator, and based on the clusters obtained from the multiple sampling and clustering and the corresponding confidence scores, it is determined whether multiple input elements in the distribution boundary region are concentrated in the same cluster. If multiple input elements located in the distribution boundary region are concentrated in the same cluster, then the operator precision deviation problem is determined to be due to an implementation defect of the problematic operator.

[0016] An embodiment of the second aspect of this application provides a neural network model transfer operator testing apparatus, comprising: The dataset acquisition module is used to acquire a first evaluation dataset containing multiple first operator input data; the first evaluation dataset refers to a dataset used to evaluate model performance in real application scenarios; The output error threshold determination module is used to determine the output error threshold based on the input data of the plurality of first operators and the first computation graph; the first computation graph represents the dependency relationship of the plurality of first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; the first model refers to the neural network model under the original framework. A test threshold determination module is used to determine multiple test thresholds based on the output error threshold. The operator output module is used to input the same set of operator input data into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result; the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the neural network model under the current framework; An error value calculation module is used to calculate the error value between the output result of the first operator and the output result of the second operator; The operator migration information determination module is used to determine the operator migration information of the second model based on the error value and the multiple test thresholds, so as to optimize the second model.

[0017] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the neural network model transfer operator testing method described in the first aspect above.

[0018] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the neural network model transfer operator testing method described in the first aspect above.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a neural network model transfer operator testing method provided in an embodiment of this application is shown. Figure 2 This illustration shows a schematic diagram of the structure of a neural network model transfer operator testing device according to an embodiment of this application; Figure 3 This application shows a schematic diagram of the structure of a computer device according to an embodiment of the present application; Figure 4 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0021] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0022] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0023] The technical scenarios involved in the embodiments of this application are described below.

[0024] With the rapid development of deep learning and large model technologies, more and more applications are adopting these technologies to achieve intelligent generation tasks. Currently, most mainstream large models are developed based on foreign deep learning frameworks such as PyTorch. Due to the need for independent control and domestic adaptation, the necessity of migrating models to domestic software and hardware and domestic AI frameworks (such as PaddlePaddle) is becoming increasingly prominent.

[0025] However, due to differences in operator implementation and underlying computational mechanisms between domestic and foreign AI frameworks, accuracy deviations often occur when migrating from foreign frameworks like PyTorch to domestic frameworks. After each migration or modification, developers need to test and locate the accuracy error to determine if it exceeds a preset threshold. When an operator is confirmed to have an accuracy anomaly, further analysis is required to determine whether the error is a normal deviation caused by numerical error or a defect (bug) in the operator implementation.

[0026] Currently, the testing and localization of operator accuracy problems still face the following difficulties: Test case selection relies on human experience: For operators with suspected problems, test cases need to be selected manually, but there is a lack of systematic methods for quickly selecting typical test cases that can reflect alignment differences. The definition of error threshold lacks reasonable criteria: there are objective errors in the output of operators between different frameworks. How to reasonably set the comparison threshold to determine whether it belongs to the implementation defect still depends on human experience. Moreover, the goal is not only to verify whether the overall accuracy meets the standard, but also to help determine whether there are real calculation errors in the operator implementation.

[0027] Based on the above, this invention proposes a precision alignment detection method, which automatically analyzes the source of error and determines whether it belongs to the operator precision problem after discovering the precision difference between frames, thereby improving the reliability and efficiency of the precision alignment process.

[0028] According to an embodiment of this application, a method for testing neural network model transfer operators is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a method for testing the transfer operator of a neural network model. Figure 1 This is a flowchart of a neural network model transfer operator testing method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the first evaluation dataset containing multiple first operator input data.

[0030] Specifically, the first evaluation dataset refers to a dataset used to evaluate model performance in real-world application scenarios, such as standard evaluation datasets like HumanEval used in code generation tasks. The model before transfer undergoes complete inference, and its operator input data is automatically collected before the operator being tested is executed. In some specific embodiments, the first evaluation dataset contains multiple first operator input data. Each first operator input data can be used as the input of the corresponding operator. Each first operator input data contains multiple input elements. For example, when the corresponding operator is a matrix multiplication operator, its corresponding first operator input data is an input matrix. The element size of the input matrix is ​​relatively large (e.g., floating-point values ​​are above ±1e4).

[0031] In some specific embodiments, before step S101 described above, the method further includes steps a1-a3: Step a1: Obtain the second assessment dataset.

[0032] Specifically, the second evaluation dataset includes multiple second operator input data.

[0033] Step a2: For any second operator input data, calculate the statistical characteristics of the second operator input data, and generate first operator input data corresponding to the second operator input data based on the statistical characteristics.

[0034] Specifically, the statistical characteristics include, but are not limited to, the mean and standard deviation (i.e., the square root of the root mean square) of the input data of the second operator.

[0035] In some specific embodiments, according to a preset test scale, corresponding first operator input data can be generated through the corresponding second operator input data and Gaussian distribution, thereby obtaining multiple first operator input data that correspond one-to-one with multiple second operator input data.

[0036] Step a3: Generate the first evaluation dataset based on the multiple first operator input data that correspond one-to-one with the multiple second operator input data.

[0037] In this embodiment of the application, by summarizing the newly generated input data of multiple first operators, a richer test sample set (i.e., the first evaluation dataset) can be formed, thereby improving the test coverage.

[0038] Step S102: Determine the output error threshold based on the input data of the plurality of first operators and the first computation graph.

[0039] Specifically, the first computation graph represents the dependency relationship of multiple first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; and the first model refers to the neural network model under the original framework.

[0040] More specifically, the inference or training process of a neural network model can be understood as a computational graph composed of a large number of operators combined and connected according to dependencies. For example, in a neural network model, operators include, but are not limited to, matrix multiplication (MatMul), convolution, softmax, and LayerNorm; each operator receives a set of input tensors (i.e., the input data of the first operator mentioned above), performs a specific operation (such as addition, multiplication, activation function, etc.), and outputs a new tensor result.

[0041] In the embodiments of this application, different frameworks (such as PyTorch, TensorFlow, ONNX, MindSpore, etc.) may have different underlying implementations of the same operator. During model migration or model optimization across frameworks, subtle differences may occur in the implementation methods and numerical computation precision of these operators, leading to inconsistencies in the final inference results. Therefore, the "operator precision testing" that this application focuses on actually involves automatically detecting the input and output behavior of each operator within the model to determine whether the output of these operators remains consistent with the results of the original framework after migration or transformation, in order to discover potential implementation defects or sources of numerical errors.

[0042] In some specific embodiments, step S102 above includes steps S1021-S1025: Step S1021: Input the input data of the plurality of first operators into the first computation graph to obtain the output result of each operator in the first computation graph.

[0043] Specifically, a set of input data (i.e., multiple first operator input data) is input into the first computation graph of the first model before frame migration. This computation graph consists of multiple operators (such as convolution, activation, normalization, etc.). The raw output of each operator when processing these data is recorded as a baseline for subsequent perturbation (error).

[0044] Step S1022: Random error terms are gradually added to the output result to generate a perturbed output result.

[0045] Specifically, by progressively adding a random error term (e.g., Gaussian noise) to the output of each operator, a perturbed output can be generated. The magnitude of the error term can be adjusted to observe the impact of different error sizes on model performance.

[0046] Step S1023: Replace the output result of the corresponding operator in the first computation graph with the perturbated output result, and obtain the model output result based on the replaced first computation graph.

[0047] Specifically, the output of the corresponding operator in the original computation graph is replaced with the perturbed output, and then subsequent calculations are performed to obtain the final model output. This step is equivalent to "simulating" how the entire model will react when the operator output is incorrect.

[0048] Step S1024: Determine the evaluation index value of the first model based on the output result of the model.

[0049] Specifically, the model output is evaluated using assessment metrics (including but not limited to accuracy, F1 score, and generated score) to obtain assessment metric values. These assessment metric values ​​reflect the performance of the first model at the current error level.

[0050] Step S1025: Adjust the magnitude of the random error term, and repeat steps S1021-S1025 until the evaluation index value is less than the preset index decrease threshold, and then determine the magnitude of the current random error term as the output error threshold.

[0051] In some specific embodiments, the preset threshold for index decline can be automatically located by statistically analyzing the signal-to-noise ratio (SNR) of multiple perturbation sample distributions or the sensitivity curve of the targeting function.

[0052] In this embodiment, the magnitude of the error term is adjusted; steps S1021-S1025 are repeated until the evaluation index value drops to a preset index decrease threshold, at which point the process stops; the magnitude of the random error term at this point is taken as the output error threshold of the operator. This embodiment gradually introduces perturbations to quantify the model's tolerance for intermediate layer output errors, thereby determining the "error threshold" for each operator.

[0053] Step S103: Determine multiple test thresholds based on the output error threshold.

[0054] Specifically, multiple test thresholds for the output error threshold can be determined according to a preset ratio. For example, when the output error threshold is... At that time, multiple test thresholds can be: , , The preset ratio can be set according to the situation, and no specific limit is made here.

[0055] Step S104: Input the same set of operator input data into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result.

[0056] Specifically, the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the model after framework migration, that is, the neural network model under the current framework.

[0057] More specifically, the same set of operator input data can be understood as several operator data points in the first evaluation dataset and / or the second evaluation dataset. In specific implementation, the same set of operator input data is input into the first computation graph to obtain the first operator output result, and the same set of operator input data is input into the second computation graph to obtain the second operator output result.

[0058] Step S105: Calculate the error value between the output result of the first operator and the output result of the second operator.

[0059] Specifically, the multiple first operators in the first computation graph correspond one-to-one with the multiple second operators in the second computation graph; the first operator outputs multiple results, the second operator outputs multiple results, and the error values ​​are multiple, each error value referring to the numerical error between the corresponding first operator output result and the corresponding second operator output result.

[0060] In some specific embodiments, the numerical error between the corresponding first operator output and the corresponding second operator output can be calculated in the following ways, including but not limited to: Error is determined element by element:

[0061] Relative error:

[0062] Statistical L2 norm error or maximum error

[0063] Step S106: Determine the operator migration information of the second model based on the error value and the plurality of test thresholds, so as to optimize the second model.

[0064] In some specific embodiments, step S106 above includes steps S1061-S1063: Step S1061: For any one of the plurality of test thresholds, compare the plurality of numerical errors with the test threshold respectively to determine the excess ratio under the test threshold.

[0065] Specifically, the excess ratio is determined based on the total number of operators and the target number of operators; the target operator refers to the operator whose numerical error is greater than the test threshold. The excess ratio can be obtained using the following formula:

[0066] in, Indicates the percentage exceeding the limit. Indicates the total number of operators. This indicates the number of operators whose numerical error is greater than the test threshold. For example, when the test threshold is... The total number of operators is 25, which is greater than the numerical error. If the number of target operators is 10, then the over-limit ratio is 10 / 25 = 0.4.

[0067] In some specific embodiments, the proportion of exceeding the limit under each test threshold can be determined in the following ways:

[0068] in, Indicates numerical error. Indicates the test threshold. =1, 1 / 2, 1 / 4, 1 / 8 Indicates the maximum test threshold; the above Indicates numerical error Greater than the test threshold The proportion of the number of operators to the total number of operators.

[0069] Step S1062: If the excess ratio is greater than a preset ratio threshold, then it is determined that the second model under the current framework has an operator precision deviation problem.

[0070] Step S1063: If the multiple over-limit ratios corresponding to the multiple test thresholds are all less than or equal to the preset ratio threshold, then it is determined that the second model under the current framework does not have an operator precision deviation problem.

[0071] Specifically, the preset ratio threshold can be manually set according to the actual situation, for example, 0.8. If the excess ratio corresponding to any of the multiple test thresholds is greater than the preset ratio threshold, it indicates that the second model under the new framework has an accuracy deviation problem. Conversely, if all excess ratios corresponding to multiple test thresholds are less than or equal to the preset ratio threshold, it indicates that the second model under the new framework does not have an accuracy deviation problem.

[0072] In some specific embodiments, after determining that the second model under the current framework does not have an operator precision deviation problem, the method further includes steps b1-b3: Step b1: For any test threshold, calculate the error distribution of all second operators at the test threshold.

[0073] Specifically, the error distribution includes the mean, variance, and out-of-limit ratio of the numerical error for each second operator. The mean is used to represent the degree of deviation of the error, the variance is used to represent the degree of fluctuation (stability) of the error, and the out-of-limit ratio is used to represent whether the numerical error is within the allowable range.

[0074] Step b2: Generate a target heatmap based on multiple error distributions corresponding to multiple test thresholds.

[0075] Specifically, the target heatmap is used to characterize the error distribution of all second operators, where the degree of exceeding the limit can be determined by the color intensity, for example, darker color = higher degree of exceeding the limit.

[0076] Step b3: Locate the defect operator using the target heatmap.

[0077] Specifically, each row of the target heatmap can represent the sequence number of the corresponding operator, and each column can be used to represent the over-limit ratio. If the over-limit ratio is greater than the preset ratio threshold, it will be directly marked in red to locate the defective operator, thereby locating the potential implementation defects or sources of inconsistency in operation.

[0078] In some specific embodiments, when it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes: For any second operator output, if the numerical error between the second operator output and the corresponding first operator output is greater than the maximum test threshold, and the element size of the first operator input data corresponding to the second operator output is greater than the preset size threshold, then it is determined that the operator accuracy deviation problem is caused by the implementation defect of the second operator corresponding to the second operator output in the second computation graph.

[0079] In this embodiment, the same operator is run in two separate frameworks, and the output differences between the two operators are compared. Generally, the output difference (numerical error) between the two operators will be relatively small and fluctuate within a reasonable range. However, if the input data still exhibits concentrated abnormal errors (much larger than other inputs) under a relatively large error threshold, it can be understood that the output error of the operator is greater than the maximum test threshold and is concentrated on the input data of operators with an element size greater than a preset size threshold. This means that the operator is not "randomly failing" but is stably deviating when dealing with a certain type of input. This phenomenon of "failure only under specific modes" usually indicates that the implementation of the operator has potential problems (bugs), such as improper handling of algorithm branches, precision overflow, or uncovered boundary conditions under specific circumstances.

[0080] Let's take matrix multiplication as an example to illustrate: Under normal circumstances, most input matrices are calculated with high accuracy, and the outputs of the two frames are almost identical. However, tests revealed that when the number of elements in the input matrix is ​​very large (e.g., floating-point values ​​above ±1e4), the operator output error suddenly increases significantly under a certain framework, and this error is concentrated on these large-value input samples.

[0081] Because the matrix multiplication implementation in the new framework uses single precision (float32), while the old framework internally used double precision (float64), when the input value is very large, the precision of float32 is insufficient, causing rounding errors to be amplified. In other words, the operator has a precision implementation defect (floating-point overflow or computational instability) in the "large value input region," and at this time it is said that "the operator has a potential implementation bug in this input region."

[0082] In some specific embodiments, each first operator input data includes multiple input elements; when it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes steps c1-c5: Step c1: For any input element in any first operator input data, calculate the position coordinates of the input element in the total element distribution of the multiple input elements; Step c2: Determine the number of input elements in the distribution boundary region based on the position coordinates of each input element, and calculate the proportion of input elements in the distribution boundary region based on the number of input elements and the total number of input elements in the first operator input data.

[0083] Specifically, the standard deviation multiple of each input element from the mean can be calculated, and the input element can be determined to be in the distribution boundary region by the standard deviation multiple. That is, if the standard deviation multiple is greater than the preset multiple threshold, the input element is determined to be in the distribution boundary region.

[0084] Step c3: If the proportion of the input element is greater than a preset proportion threshold, then the second operator corresponding to the input data of the first operator in the second calculation graph is initially determined as the problem operator.

[0085] Specifically, when the input elements in the distribution boundary region account for a preset percentage threshold of the total input elements, it indicates that the operator has a stability problem near the extreme input, and thus it can be preliminarily identified as a problematic operator.

[0086] Step c4: Perform multiple sampling clustering on all input elements in the input data of the first operator, and determine whether multiple input elements in the distribution boundary region are concentrated in the same cluster based on the clusters obtained from the multiple sampling clustering and the corresponding confidence scores.

[0087] Specifically, the method identifies "stable and invariant" boundary cluster structures through multiple independent clustering operations, rather than relying on random artifacts that may be generated by a single clustering. First, all input elements are sampled and clustered multiple times. Second, based on the clusters obtained from the multiple results and their corresponding confidence levels, it is determined whether the boundary elements are concentrated in the same cluster.

[0088] In some specific embodiments, K-means, DBSCAN, or simple histogram methods can also be used to cluster multiple input elements in the input data of the first operator; the input elements located in the distribution boundary region are marked to determine whether these input elements are concentrated in the same cluster.

[0089] Step c5: If multiple input elements in the distribution boundary region are concentrated in the same cluster, then it is determined that the operator precision deviation problem is caused by the implementation defect of the problematic operator.

[0090] Specifically, if these input elements are concentrated in the same cluster or distributed in a high-density area, it means that the operator has a systematic bias under this type of input feature, which may indicate an implementation defect.

[0091] The embodiments of this application can automatically complete the detection, distribution analysis, and potential defect location of operator-level precision errors without manual intervention. This method significantly improves the efficiency and accuracy analysis capabilities of operator testing during cross-framework migration, providing strong technical support for AI framework migration and verification.

[0092] Corresponding to the above implementation methods for testing neural network model transfer operators, this application also provides a neural network model transfer operator testing apparatus for executing the neural network model transfer operator testing method described in any of the above embodiments. Figure 2 As shown, the test apparatus for the neural network model transfer operator includes: The dataset acquisition module is used to acquire a first evaluation dataset containing multiple first operator input data; the first evaluation dataset refers to a dataset used to evaluate model performance in real application scenarios; The output error threshold determination module is used to determine the output error threshold based on the input data of the plurality of first operators and the first computation graph; the first computation graph represents the dependency relationship of the plurality of first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; the first model refers to the neural network model under the original framework. A test threshold determination module is used to determine multiple test thresholds based on the output error threshold. The operator output module is used to input the same set of operator input data into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result; the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the neural network model under the current framework; An error value calculation module is used to calculate the error value between the output result of the first operator and the output result of the second operator; The operator migration information determination module is used to determine the operator migration information of the second model based on the error value and the multiple test thresholds, so as to optimize the second model.

[0093] Optionally, a first assessment dataset generation module is used to obtain a second assessment dataset; the second assessment dataset includes multiple second operator input data; for any second operator input data, the statistical characteristics of the second operator input data are calculated, and first operator input data corresponding to the second operator input data is generated according to the statistical characteristics; the first assessment dataset is generated according to the multiple first operator input data that correspond one-to-one with the multiple second operator input data.

[0094] Optionally, the output error threshold determination module is further configured to: input the input data of the plurality of first operators into the first computation graph to obtain the output result of each operator in the first computation graph; progressively add random error terms to the output results to generate perturbed output results; replace the output results of the corresponding operators in the first computation graph with the perturbed output results, and obtain the model output results based on the replaced first computation graph; determine the first model evaluation index value based on the model output results; adjust the magnitude of the random error terms, and repeatedly execute the steps of inputting the input data of the plurality of first operators into the first computation graph to obtain the output results of each operator in the first computation graph; progressively adding random error terms to the output results to generate perturbed output results, until the evaluation index value is less than a preset index decrease threshold, and determine the magnitude of the current random error term as the output error threshold.

[0095] Optionally, the operator migration information determination module is further configured to compare multiple numerical errors with any one of the multiple test thresholds to determine the out-of-limit ratio under the test threshold; the out-of-limit ratio is determined based on the total number of operators and the target number of operators; the target operator refers to the operator whose numerical error is greater than the test threshold; if the out-of-limit ratio is greater than a preset ratio threshold, it is determined that the second model under the current framework has an operator precision deviation problem; if the multiple out-of-limit ratios corresponding one-to-one with the multiple test thresholds are all less than or equal to the preset ratio threshold, it is determined that the second model under the current framework does not have an operator precision deviation problem.

[0096] Optionally, a defect operator localization module is used to calculate the error distribution of all second operators under any test threshold; the error distribution includes the mean, variance, and out-of-limit ratio of the numerical error of each second operator; generate a target heatmap based on multiple error distributions corresponding to multiple test thresholds; the target heatmap is used to characterize the error distribution of all second operators; and locate the defect operator through the target heatmap.

[0097] Optionally, the device further includes: an operator positioning module, configured to calculate the position coordinates of any input element in the total element distribution of the plurality of input elements for any input element in any first operator input data; determine the number of input elements in the distribution boundary region based on the position coordinates of each input element, and calculate the proportion of input elements in the distribution boundary region based on the number of input elements and the total number of input elements in the first operator input data; if the proportion of input elements is greater than a preset proportion threshold, then preliminarily identify the second operator corresponding to the first operator input data in the second calculation graph as a problematic operator; cluster the plurality of input elements in the first operator input data, and determine whether the plurality of input elements in the distribution boundary region are concentrated in the same cluster; if the plurality of input elements in the distribution boundary region are concentrated in the same cluster, then determine that the operator accuracy deviation problem is caused by the implementation defect of the problematic operator.

[0098] The neural network model transfer operator testing device and the neural network model transfer operator testing method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods used, run or implemented by the applications stored therein.

[0099] This application also provides a computer device for performing the above-described neural network model transfer operator testing method. Please refer to... Figure 3 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 3 As shown, the computer device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected through the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the neural network model transfer operator testing method provided in any of the foregoing embodiments of this application.

[0100] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0101] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The neural network model transfer operator testing method disclosed in any of the foregoing embodiments can be applied to the processor 300, or implemented by the processor 300.

[0102] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0103] The computer device provided in this application embodiment and the neural network model transfer operator testing method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0104] This application also provides a computer-readable storage medium corresponding to the neural network model transfer operator testing method provided in the foregoing embodiments. Please refer to... Figure 4 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the neural network model transfer operator testing method provided in any of the foregoing embodiments.

[0105] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0106] The computer-readable storage medium provided in the above embodiments of this application and the neural network model transfer operator testing method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0107] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0108] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0109] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0110] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for testing transfer operators in neural network models, characterized in that, The method includes: Obtain a first evaluation dataset containing multiple first operator input data; the first evaluation dataset refers to a dataset used to evaluate model performance in real application scenarios; The output error threshold is determined based on the input data of the plurality of first operators and the first computation graph; the first computation graph represents the dependency relationship of the plurality of first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; the first model refers to the neural network model under the original framework; Multiple test thresholds are determined based on the output error threshold; The same set of operator input data is input into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result; the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the neural network model under the current framework; Calculate the error value between the output of the first operator and the output of the second operator; The operator migration information of the second model is determined based on the error value and the plurality of test thresholds in order to optimize the second model.

2. The method according to claim 1, characterized in that, Before obtaining a first evaluation dataset containing multiple first operator input data, the method further includes: Obtain the second evaluation dataset; the second evaluation dataset includes multiple input data for the second operator; For any second operator input data, calculate the statistical characteristics of the second operator input data, and generate first operator input data corresponding to the second operator input data based on the statistical characteristics; The first evaluation dataset is generated based on the multiple first operator input data that correspond one-to-one with the multiple second operator input data.

3. The method according to claim 1 or 2, characterized in that, Determining the output error threshold based on the input data of the plurality of first operators and the first computation graph includes: The input data of the plurality of first operators are input into the first computation graph to obtain the output result of each operator in the first computation graph; Random error terms are gradually added to the output to generate a perturbed output. The perturbated output result replaces the output result of the corresponding operator in the first computation graph, and the model output result is obtained based on the replaced first computation graph; The evaluation index value of the first model is determined based on the output result of the model. Adjust the magnitude of the random error term, repeatedly input the input data of the multiple first operators into the first computation graph, and obtain the output result of each operator in the first computation graph; gradually add random error terms to the output results to generate perturbed output results, until the evaluation index value is less than the preset index decrease threshold, and then determine the magnitude of the current random error term as the output error threshold.

4. The method according to claim 1 or 2, characterized in that, The multiple first operators in the first computation graph correspond one-to-one with the multiple second operators in the second computation graph; the first operator outputs multiple results, the second operator outputs multiple results, and the error values ​​are multiple, each error value referring to the numerical error between the corresponding first operator output result and the corresponding second operator output result; Determining the operator transfer information of the second model based on the error value and the plurality of test thresholds includes: For any one of the plurality of test thresholds, multiple numerical errors are compared with the test threshold to determine the excess ratio under the test threshold; the excess ratio is determined based on the total number of operators and the target number of operators; the target operator refers to the operator whose numerical error is greater than the test threshold. If the excess ratio is greater than a preset ratio threshold, then it is determined that the second model under the current framework has an operator precision deviation problem; If the multiple out-of-limit ratios corresponding to the multiple test thresholds are all less than or equal to the preset ratio threshold, then it is determined that the second model under the current framework does not have an operator precision deviation problem.

5. The method according to claim 4, characterized in that, After determining that the second model under the current framework does not have an operator precision deviation problem, the method further includes: For any given test threshold, calculate the error distribution of all second operators at that test threshold; the error distribution includes the mean, variance, and out-of-limit percentage of the numerical error for each second operator; A target heatmap is generated based on multiple error distributions corresponding to multiple test thresholds; the target heatmap is used to characterize the error distribution of all second operators; The defect operator is located using the target heatmap.

6. The method according to claim 4, characterized in that, When it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes: For any second operator output, if the numerical error between the second operator output and the corresponding first operator output is greater than the maximum test threshold, and the element size of the first operator input data corresponding to the second operator output is greater than the preset size threshold, then it is determined that the operator accuracy deviation problem is caused by the implementation defect of the second operator corresponding to the second operator output in the second computation graph.

7. The method according to claim 4, characterized in that, Each first operator input data includes multiple input elements; when it is determined that the second model under the current framework has an operator precision deviation problem, the method further includes: For any input element in any first operator input data, calculate the position coordinates of the input element in the total element distribution of the plurality of input elements; The number of input elements in the distribution boundary region is determined based on the position coordinates of each input element, and the proportion of input elements in the distribution boundary region is calculated based on the number of input elements and the total number of input elements in the first operator input data. If the proportion of the input element is greater than a preset proportion threshold, then the second operator in the second computation graph corresponding to the input data of the first operator is initially determined as the problem operator; Multiple sampling and clustering are performed on all input elements in the input data of the first operator, and based on the clusters obtained from the multiple sampling and clustering and the corresponding confidence scores, it is determined whether multiple input elements in the distribution boundary region are concentrated in the same cluster. If multiple input elements located in the distribution boundary region are concentrated in the same cluster, then the operator precision deviation problem is determined to be due to an implementation defect of the problematic operator.

8. A neural network model transfer operator testing device, characterized in that, The device includes: The dataset acquisition module is used to acquire a first evaluation dataset containing multiple first operator input data; the first evaluation dataset refers to a dataset used to evaluate model performance in real application scenarios; The output error threshold determination module is used to determine the output error threshold based on the input data of the plurality of first operators and the first computation graph; the first computation graph represents the dependency relationship of the plurality of first operators in the first model; the first operator refers to the smallest functional unit that performs logical operations in the first computation graph; the first model refers to the neural network model under the original framework; A test threshold determination module is used to determine multiple test thresholds based on the output error threshold. The operator output module is used to input the same set of operator input data into the first computation graph and the second computation graph respectively to obtain the first operator output result and the second operator output result; the second computation graph represents the dependency relationship of the multiple first operators in the second model; the second model refers to the neural network model under the current framework; An error value calculation module is used to calculate the error value between the output result of the first operator and the output result of the second operator; The operator migration information determination module is used to determine the operator migration information of the second model based on the error value and the multiple test thresholds, so as to optimize the second model.

9. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the neural network model transfer operator testing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the neural network model transfer operator testing method according to any one of claims 1 to 7.

Images