Operator non-uniformity degree evaluation method, electronic device, and storage medium

Abstract

The application provides a non-uniformity degree evaluation method of an operator, an electronic device and a storage medium, and the method comprises the following steps: obtaining a reference output of the operator and a plurality of groups of test outputs obtained by running the operator multiple times; comparing each group of test outputs with the reference output by taking a tensor as a comparison unit to obtain a non-uniformity score of each test tensor in each group of test outputs compared with a reference tensor in the reference output, wherein the non-uniformity score indicates whether each element in the test tensor and the reference tensor is consistent; and evaluating the non-uniformity degree of the operator based on the non-uniformity score of each test tensor in each group of test outputs. The method, the electronic device and the storage medium provided by the application realize the quantification of the non-uniformity of the operator, thereby providing a basis for comparing the non-uniformity degrees of different versions of the operator, helping to accurately locate the stability problem of the artificial intelligence model, and improving the transplantation efficiency of the artificial intelligence model adapted to the chip.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a non-consistency degree evaluation method of an operator, an electronic device and a storage medium. BACKGROUND

[0002] Consistency evaluation is an important means to measure the stability of an operator. The implementation is to apply a fixed input to the operator and execute the operator multiple times to determine whether the outputs are completely the same. If there is at least one output that is inconsistent with the outputs of other rounds in the multiple execution process, the operator is determined to be a non-consistent operator.

[0003] However, the current consistency evaluation method for operators has obvious limitations. Specifically, it can only qualitatively determine whether the operator has consistency, and cannot further quantify the degree of non-consistency of the operator. In fact, the degree of non-consistency directly affects the size of the random fluctuations caused by the application of the operator in the artificial intelligence model, and further affects the accuracy and stability of the artificial intelligence model.

[0004] Therefore, how to realize the quantification of the non-consistency degree of the operator is still a problem to be solved in the field. SUMMARY

[0005] The present application provides a non-consistency degree evaluation method of an operator, an electronic device and a storage medium to solve the defect that only qualitative analysis is performed on whether the operator has consistency in the related art, and the actual application requirements cannot be met.

[0006] The present application provides a non-consistency degree evaluation method of an operator, comprising:

[0007] obtaining a reference output of an operator and multiple sets of test outputs obtained by multiple running of the operator;

[0008] comparing each set of test outputs with the reference output respectively as a comparison unit to obtain a non-consistency score of each test tensor in each set of test outputs compared with a reference tensor in the reference output, the non-consistency score indicating whether each element in the test tensor and the reference tensor is consistent;

[0009] based on the non-consistency score of each test tensor in each set of test outputs, evaluating the non-consistency degree of the operator.

[0010] According to the non-consistency degree evaluation method of the operator provided by the present application, the comparison of each set of test outputs with the reference output as a comparison unit to obtain a non-consistency score of each test tensor in each set of test outputs compared with a reference tensor in the reference output comprises:

[0011] comparing each test tensor in each set of test outputs with the base tensor as a comparison unit to obtain an element comparison result of each test tensor in each set of test outputs compared with the base tensor;

[0012] For each test tensor in each set of test outputs, based on the element comparison result, a comparison score of each element in the test tensor is determined, and based on the position and comparison score of each element in the test tensor, an inconsistency score of the test tensor is determined.

[0013] According to the application, an operator inconsistency degree evaluation method is provided, and the inconsistency degree of the operator is evaluated based on the inconsistency score of each test tensor in each set of test outputs, which comprises:

[0014] Based on the inconsistency score of the same test tensor in each set of test outputs, the inconsistency degree of the test tensor is evaluated;

[0015] Based on the inconsistency degree of each test tensor, the inconsistency degree of the operator is determined.

[0016] According to the application, an operator inconsistency degree evaluation method is provided, and the inconsistency degree of the test tensor is evaluated based on the inconsistency score of the same test tensor in each set of test outputs, which comprises:

[0017] Based on the number of elements of the test tensor, the number of running times of the operator, and the inconsistency score of the test tensor in each set of test outputs, the observed consistency probability and the expected consistency probability of the test tensor are determined;

[0018] Based on the observed consistency probability and the expected consistency probability of the test tensor, the inconsistency degree of the test tensor is determined.

[0019] According to the application, an operator inconsistency degree evaluation method is provided, and the observed consistency probability and the expected consistency probability of the test tensor are determined based on the number of elements of the test tensor, the number of running times of the operator, and the inconsistency score of the test tensor in each set of test outputs, which comprises:

[0020] Based on the inconsistency score of the test tensor in each set of test outputs, the number of times that each element in the test tensor belongs to various score types in the inconsistency score is counted;

[0021] Based on the number of times that each element belongs to various score types in the inconsistency score, and the number of elements and the number of running times, the observed consistency probability and the expected consistency probability of the test tensor are determined.

[0022] The operator inconsistency degree evaluation method provided by the application further comprises:

[0023] The version comparison is performed based on the inconsistency degrees of the operator in the multiple versions.

[0024] The version comparison is performed based on the inconsistency degrees of the operator in the multiple versions.

[0025] The version comparison is performed based on the inconsistency degrees of each test tensor of the operator in the multiple versions and the importance of the test tensor.

[0026] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the operator inconsistency degree evaluation method according to any one of the above when executing the program.

[0027] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program is executable on a processor to implement the operator inconsistency degree evaluation method according to any one of the above.

[0028] The application further provides a computer program product comprising a computer program, and the computer program is executable on a processor to implement the operator inconsistency degree evaluation method according to any one of the above.

[0029] The operator inconsistency degree evaluation method, the electronic device, and the storage medium provided by the application can obtain the inconsistency scores of each test tensor in the test output by comparing each group of test output and reference output, thereby reflecting whether each element in the test tensor and the reference tensor is consistent or not, and evaluating the inconsistency degree of the operator on this basis, thereby realizing the quantification of the operator inconsistency, providing a basis for comparing the inconsistency degrees of different versions of the operator, and helping to accurately locate the stability problem of the artificial intelligence model and improving the porting efficiency of the artificial intelligence model to the chip. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0031] Figure 1 is one of the flowcharts of the operator inconsistency degree evaluation method provided by the application;

[0032] Figure 2 is one of the comparison diagrams of the test tensor and the reference tensor provided by the present application;

[0033] Figure 3 is one of the comparison diagrams of the test tensor and the reference tensor provided by the present application;

[0034] Figure 4 is one of the comparison diagrams of the test tensor and the reference tensor provided by the present application;

[0035] Figure 5 is a version comparison diagram provided by the present application;

[0036] Figure 6 is a flowchart of the non-uniformity degree evaluation method of the operator provided by the present application;

[0037] Figure 7 is a structural diagram of the non-uniformity degree evaluation device of the operator provided by the present application;

[0038] Figure 8 is a structural diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] In the field of artificial intelligence technology, the evaluation of the pros and cons of an operator is a crucial step, which directly relates to the performance and accuracy of the artificial intelligence model. When evaluating an operator, multiple dimensions are usually considered, including but not limited to performance, accuracy, and consistency.

[0041] Among them, consistency evaluation is an important means to measure the stability of the operator. The implementation mainly applies a fixed input to the operator and executes it multiple times to determine whether the output is exactly the same. If there is at least one output that is inconsistent with other rounds in the multiple execution process, the operator is determined to be a non-uniform operator.

[0042] However, the current evaluation method for the consistency of the operator has obvious limitations. Specifically, it can only qualitatively determine whether the operator has consistency, in other words, the current evaluation for the consistency of the operator is only used to determine whether the operator has consistency, which is a qualitative evaluation of the operator. This evaluation index is too simple, lacks quantitative evaluation, and cannot evaluate or compare the advantages and disadvantages of multiple versions of the operator when there are multiple versions of the operator. It can be understood that the multiple versions of the operator referred to here refer to different implementation ways of the operator, which can specifically be manifested as different splitting dimensions, different splitting granularities, etc.

[0043] Specifically, this qualitative evaluation method is not sufficient when facing complex and variable artificial intelligence application scenarios. In particular, in an artificial intelligence model, although some non-consistent operators may have similar performance to other operators, the random fluctuations in their output may have a serious impact on the overall stability of the model. Here, random fluctuations refer to the case where a non-consistent operator outputs inconsistently and fluctuates in multiple executions for the same input. The random fluctuation value is the specific difference data generated due to random fluctuations.

[0044] For example, for two implementation versions A and B of an operator, if version A is comparable in performance to version B, but the output of version A contains more random fluctuations than the output of version B, such fluctuations may cause a lot of adverse interference to the artificial intelligence model. Since the current evaluation for the consistency of the operator is mostly a qualitative evaluation method, assuming that both version A and version B are evaluated as non-consistent operators, researchers cannot determine whether to apply version A or version B in the artificial intelligence model based on the qualitative consistency evaluation. However, the fact that the output of version A contains more random fluctuations than the output of version B can actually be reflected by a quantitative consistency evaluation method, that is, if the degree of non-consistency of the operator can be quantified, the quantified degree of non-consistency can provide an effective reference for applying version A or version B in the artificial intelligence model.

[0045] Therefore, how to quantify the degree of non-consistency of the operator is still a problem to be solved in the field.

[0046] To solve this problem, the present application provides a method for evaluating the degree of non-consistency of an operator. Figure 1 is one of the flowcharts of the method for evaluating the degree of non-consistency of the operator provided by the present application, as shown in Figure 1 The method comprises the following steps.

[0047] Step 110: obtaining the benchmark output of the operator and multiple sets of test outputs obtained by multiple runs of the operator.

[0048] The operator here is the operator that needs to be evaluated for the degree of non-consistency.

[0049] The reference output and the test output of the operator are both outputs obtained by running the operator, and the reference output and the test output are specifically obtained in the process of repeatedly running the operator under the same conditions for multiple times. The same conditions referred to herein include that the running environment of the operator is the same, that is, the hardware device running the operator is the same, or the performance parameters of the hardware device running the operator are the same, and also include that the input of the operator is the same when the operator is running.

[0050] For example, the operator can be repeatedly run under the same conditions for multiple times, the output obtained for the first time is taken as the reference output, and the output obtained for each time thereafter is taken as the test output; or the operator can be repeatedly run under the same conditions for multiple times, the output obtained for any one time is taken as the reference output, and the output obtained for each of the remaining times is taken as the test output.

[0051] It can be understood that the reference output is an output that serves as a Golden for comparison when the degree of non-uniformity of the operator is evaluated, and the test output is an output that is compared with the reference output when the degree of non-uniformity of the operator is evaluated. After obtaining the reference output and the test output of the operator, the degree of non-uniformity of the operator can be evaluated based thereon.

[0052] In step 120, each set of test output is compared with the reference output respectively by taking a tensor as a comparison unit, and a non-uniformity score of each test tensor in each set of test output compared with a reference tensor in the reference output is obtained, the non-uniformity score indicating whether each element in the test tensor and the reference tensor is uniform.

[0053] Specifically, the output of the operator can include one or more tensors. In the test output, the above-mentioned tensors are recorded as test tensors, and in the reference output, the above-mentioned tensors are recorded as reference tensors. It can be understood that the number and type of test tensors in the test output and the number and type of reference tensors in the reference output are the same, that is, the test tensors in the test output and the reference tensors in the reference output correspond one by one.

[0054] Therefore, for each set of test output, each test tensor in the test output can be compared with the corresponding reference tensor in the reference output by taking a tensor as a comparison unit.

[0055] In the embodiment of the present application, in the process of comparing the test tensor and the reference tensor one by one, the score can be regarded as a score of the inconsistency of the test tensor compared with the reference tensor, and thus the result of comparing the test tensor and the reference tensor is taken as the inconsistency score of the test tensor compared with the reference tensor. Here, for each set of test outputs, each test tensor in the test output can obtain a corresponding inconsistency score, and the inconsistency score can reflect whether each element in the test tensor is consistent with the element at the corresponding position in the reference tensor.

[0056] Specifically, for the test tensor and the corresponding reference tensor, the information reflecting the inconsistency between the two includes the number of inconsistent elements in the test tensor and the reference tensor, the position of the inconsistent elements in the test tensor and the reference tensor, and the inconsistency degree of the inconsistent elements in the test tensor and the reference tensor.

[0057] For example, Figure 2 is one of the comparison diagrams of the test tensor and the reference tensor provided by the present application, as shown in Figure 2 assuming that the reference tensor and the test tensors 1 and 2 are all 3x3 matrices, comparing the reference tensor with the test tensor 1 can find that the test tensor 1 is inconsistent with the reference tensor at the two coordinate positions (2, 1) and (2, 2); comparing the reference tensor with the test tensor 2 can find that the test tensor 2 is inconsistent with the reference tensor at the two coordinate positions (1, 1) and (3, 3), and thus it can be seen that the positions of the inconsistent elements may be different for different test tensors compared with the reference tensor.

[0058] For example, Figure 3 is the second comparison diagram of the test tensor and the reference tensor provided by the present application, as shown in Figure 3 assuming that the reference tensor and the test tensors 1 and 2 are all 3x3 matrices, comparing the reference tensor with the test tensor 1 can find that the test tensor 1 is inconsistent with the reference tensor at one element; comparing the reference tensor with the test tensor 2 can find that the test tensor 2 is inconsistent with the reference tensor at two elements, and thus it can be seen that the number of inconsistent elements may be different for different test tensors compared with the reference tensor.

[0059] For example, Figure 4 is the third comparison diagram of the test tensor and the reference tensor provided by the present application, as shown in Figure 4As shown, assuming that the reference tensor and the test tensors 1 and 2 are all 3x3 matrices, comparing the reference tensor with the test tensor 1 can show that the number of elements of the test tensor 1 that are inconsistent with the reference tensor is 9; comparing the reference tensor with the test tensor 2 can show that the number of elements of the test tensor 2 that are inconsistent with the reference tensor is 5.2. It can be seen that the number of inconsistent elements of different test tensors compared with the reference tensor can be different, and the magnitude of the inconsistency reflected by the number of inconsistent elements can also be different.

[0060] The inconsistency score reflecting whether each element in the test tensor is consistent with the element at the corresponding position in the reference tensor can cover the inconsistency degree embodied in at least one of the position, number and value of the inconsistent elements. For example, the inconsistency score can be represented as a tensor obtained by subtracting the reference tensor from the test tensor, or the inconsistency score can be represented as a tensor of the same size as the test tensor and the reference tensor, where the value of each element is used to reflect whether the element at the corresponding position in the test tensor and the reference tensor is consistent. The embodiments of the present application do not make specific limitations on this.

[0061] Step 130, based on the inconsistency score of each test tensor in each group of test outputs, evaluating the inconsistency degree of the operator.

[0062] Specifically, after obtaining the inconsistency score of each test tensor in each group of test outputs, the inconsistency degree of the operator can be evaluated based on this. Here, when evaluating the inconsistency degree of the operator, considering that the inconsistency degrees of different tensors of the output can be different, the inconsistency degree of the operator can be specifically represented as the inconsistency degree of each tensor in the output of the operator.

[0063] Therefore, for any one tensor, the inconsistency scores of the tensor in each group of test outputs can be counted, and the inconsistency degree of the tensor can be calculated based on this. For example, the kappa coefficient of the inconsistency scores of the tensor in each group of test outputs can be calculated as the inconsistency degree of the tensor; for another example, the Krippendorff’s Alpha parameter of the inconsistency scores of the tensor in each group of test outputs can be calculated as the inconsistency degree of the tensor. The embodiments of the present application do not make specific limitations on this. It can be understood that the kappa coefficient and the Krippendorff’s Alpha parameter here are both indexes for consistency test.

[0064] In the method provided in the embodiments of the present application, the inconsistency scores of each test tensor in the test output can be obtained by comparing each set of test output and the reference output, so as to reflect whether each element in the test tensor and the reference tensor is consistent or not, and on this basis, the inconsistency degree of the operator is evaluated, thereby realizing the quantification of the operator inconsistency, thereby providing a basis for comparing the inconsistency degrees of different versions of the operator, and helping to accurately locate the stability problem of the artificial intelligence model and improving the transplantation efficiency of the artificial intelligence model adapted to the chip.

[0065] Based on the above embodiments, in step 120, the test output and the reference output are compared in the tensor as the comparison unit, and the inconsistency score of each test tensor in each set of test output compared with the reference tensor in the reference output is obtained, including:

[0066] The test output and the reference output are compared in the tensor as the comparison unit, and the element comparison result of each test tensor in each set of test output compared with the reference tensor is obtained.

[0067] For each test tensor in each set of test output, the comparison score of each element in the test tensor is determined based on the element comparison result, and the inconsistency score of the test tensor is determined based on the position and comparison score of each element in the test tensor.

[0068] Specifically, in the comparison process in the tensor as the comparison unit, for each test tensor in each set of test output, the element comparison result of the test tensor compared with the corresponding reference tensor can be obtained. Here, the element comparison result can include the comparison result of each element in the test tensor compared with the corresponding element in the reference tensor, and specifically can be whether each element in the test tensor is consistent with the corresponding element in the reference tensor.

[0069] For each test tensor, the comparison score of each element in the test tensor can be assigned based on the element comparison result, for example, for the element judged as consistent in the element comparison result, the comparison score of the element can be assigned as 1, and for the element judged as inconsistent in the element comparison result, the comparison score of the element can be assigned as 0.

[0070] After obtaining the comparison score of each element in the test tensor, the inconsistency score which can reflect whether each element in the test tensor and the reference tensor is consistent or not can be constructed in combination with the comparison score of each element and the position of each element in the test tensor.

[0071] Here, the inconsistency score can be in the form of a tensor, the size of the inconsistency score can be consistent with the size of the test tensor, and the elements of the inconsistency score take the comparison scores of the elements at the corresponding positions in the test tensor. For example, for the test tensor 1 in Figure 2 , the comparison scores of the test tensor 1 at positions (2, 1) and (2, 2) are 0, and the comparison scores at other positions are 1, so the inconsistency score can be in the form of

[0072]

[0073] Alternatively, the inconsistency score can also be in the form of a list, specifically a mapping relationship table between the coordinate values of each element in the test tensor and the comparison scores of the elements, and the comparison score of the element at the position corresponding to the coordinate value can be obtained by looking up the coordinate value. For example, for the test tensor 1 in Figure 2 , the comparison scores of the test tensor 1 at positions (2, 1) and (2, 2) are 0, and the comparison scores at other positions are 1; for the test tensor 2 in Figure 2 , the comparison scores of the test tensor 2 at positions (1, 1) and (3, 3) are 0, and the comparison scores at other positions are 1. Thus, the inconsistency scores of the test tensor 1 and the test tensor 2 can be obtained in the form of the following table:

[0074]

[0075] Based on any of the above embodiments, in step 130, the inconsistency degree of the operator is evaluated based on the inconsistency scores of each test tensor in each set of test outputs, including:

[0076] The inconsistency degree of the test tensor is evaluated based on the inconsistency scores of the same test tensor in each set of test outputs;

[0077] The inconsistency degree of the operator is determined based on the inconsistency degrees of each test tensor.

[0078] Specifically, considering that the inconsistency degrees of different tensors in the output can be different, the inconsistency degree of the operator can specifically represent the inconsistency degrees of each tensor in the output of the operator.

[0079] Thus, for any tensor, the inconsistency scores of the tensor in each set of test outputs can be counted, and the inconsistency degree of the tensor can be calculated based thereon.

[0080] For example, for a test tensor, the non-consistency scores of the test tensor in each group of test outputs can be taken as evaluation objects, and the Kappa coefficient between the evaluation objects is calculated to measure the consistency between the evaluation objects, and thus the calculated Kappa coefficient can be taken as the non-consistency degree of the test tensor.

[0081] For example, for a test tensor, Krippendorff's Alpha parameter can be applied as the non-consistency degree of the test tensor. Specifically, Krippendorff's Alpha parameter is widely used in content analysis, text coding, survey data coding and other fields to measure the consistency degree of different coders coding the same group of data. Based on the above idea, in the process of taking Krippendorff's Alpha parameter as the non-consistency degree of the test tensor, each element of the test tensor in a group of test outputs can be regarded as a text segment, each group of test outputs can be regarded as a coder, and whether the elements are consistent can be regarded as a coding result, that is, each running of the operator can be understood as a test, and thus a coding result can be generated in each test, that is, the non-consistency score of the test tensor in the test output obtained when each running of the operator. Thus, the non-consistency scores of the test tensor in each group of test outputs can be taken as the coding results generated in each test to calculate Krippendorff's Alpha parameter, and the calculated Krippendorff's Alpha parameter can be taken as the non-consistency degree of the test tensor.

[0082] After obtaining the non-consistency degree of each test tensor, the non-consistency degrees of each test tensor can be summarized as the non-consistency degree of the operator.

[0083] Based on any of the above embodiments, in step 130, the non-consistency degree of the test tensor is evaluated based on the non-consistency scores of the same test tensor in each group of test outputs, including:

[0084] Based on the number of elements of the test tensor, the number of running times of the operator, and the non-consistency scores of the test tensor in each group of test outputs, the observed consistency probability and the expected consistency probability of the test tensor are determined;

[0085] Based on the observed consistency probability and the expected consistency probability of the test tensor, the non-consistency degree of the test tensor is determined.

[0086] Specifically, when evaluating the non-consistency degree of a test tensor, Kappa coefficient calculation can be used. Here, the calculation of Kappa coefficient can be based on the observed consistency probability and the expected consistency probability.

[0087] wherein the observed consistency probability is generally represented as The observed consistency probability refers to the consistency degree obtained by observation on the basis of the non-consistency scores of the test tensor in each group of test outputs as evaluation objects. The observed consistency probability reflects the closeness of the non-consistency scores in each group of test outputs in the classification scoring of whether the elements are consistent. The observed consistency probability can be calculated from the number of elements of the test tensor, the number of running times of the operator, and the non-consistency scores of the test tensor in each group of test outputs. Here, the number of running times of the operator is the number of groups of test outputs.

[0088] The expected consistency probability is generally represented as The expected consistency probability refers to the probability that the non-consistency scores of the test tensor in each group of test outputs are consistent due to randomness without considering the non-consistency scores of the test tensor in each group of test outputs. The expected consistency probability can reflect the possibility that the non-consistency scores in each group of test outputs are consistent in the case of completely random classification scoring of whether the elements are consistent. The expected consistency probability can be calculated from the number of elements of the test tensor, the number of running times of the operator, and the non-consistency scores of the test tensor in each group of test outputs. Here, the number of running times of the operator is the number of groups of test outputs.

[0089] After obtaining the observed consistency probability and the expected consistency probability, the non-consistency degree of the test tensor can be calculated therefrom, for example, can be represented as the following formula:

[0090]

[0091] In the formula, is the non-consistency degree of the test tensor.

[0092] Based on any one of the above embodiments, in step 130, the observed consistency probability and the expected consistency probability of the test tensor are determined based on the number of elements of the test tensor, the number of running times of the operator, and the non-consistency scores of the test tensor in each group of test outputs, including:

[0093] Based on the non-consistency scores of the test tensor in each group of test outputs, the number of times that each element in the test tensor belongs to various score types in the non-consistency scores is counted;

[0094] Based on the number of times that each element belongs to various score types in the non-consistency scores, and the number of elements and the number of running times, the observed consistency probability and the expected consistency probability of the test tensor are determined.

[0095] Specifically, the inconsistency score of the test tensor in any set of test outputs can include the score type of each element in the test tensor in that set of test outputs. For example, a score of 1 indicates consistency, and a score of 0 indicates inconsistency, so the score type can include both 1 and 0. Therefore, after obtaining the inconsistency score of the test tensor in each set of test outputs, we can statistically analyze the score type of each element in the test tensor in each set of test outputs, thus obtaining the number of times each element belongs to each score type in the inconsistency score. For example, in a total of 1000 sets of test outputs, an element belongs to score type 1 900 times and to score type 0 100 times.

[0096] Based on this, the observed consistency probability and expected consistency probability can be calculated for the test tensor based on the number of times each element belongs to various score types in the inconsistent score, as well as the number of elements and the number of runs mentioned above.

[0097] The calculation method for the probability of observational consistency can be expressed as follows:

[0098]

[0099] In the formula, The number of times the operator is run. For the index of an element in the test tensor, For indexing score types in inconsistent scores, The total number of categories that score an element. . For the first The number of times an element is classified into the j-th class, for example For the first The number of times each element has a score of type 1. For the first The number of times each element's score type is 0. This is calculated from... For the first The probability of observational agreement for each element.

[0100] Based on this, the observational consistency probability of the test tensor can be calculated using the following formula:

[0101]

[0102] In the formula, To test the observation consistency probability of the tensor To test the total number of elements in a tensor, The value of depends on the specific dimensions of the test tensor. For example, if the shape of the test tensor is [batch, height, width], then... .

[0103] The calculation method of the expected consistency probability can be expressed as follows:

[0104]

[0105]

[0106] In the formula, is the expected consistency probability of the jth score type. is the expected consistency probability of the test tensor.

[0107] Based on any of the above embodiments, the method further comprises:

[0108] Comparing the versions based on the degree of inconsistency of the operator in the multiple versions.

[0109] Specifically, for the case where there are multiple implementation versions of an operator, each version of the operator can be regarded as an independent operator, and for each version of the operator, the above-mentioned inconsistency degree evaluation method is performed respectively, thereby obtaining the inconsistency degree of each version of the operator. That is, the inconsistency degree of the operator in each version.

[0110] In this case, the comparison of different versions of the same operator can be realized by comparing the inconsistency degree of the operator in each version, thereby providing a condition for selecting the version of the operator to be applied in the artificial intelligence model.

[0111] Here, the comparison of different versions of the operator can be performed by comparing the inconsistency degree of the same test tensor in different versions. For example, Figure 5 is a version comparison diagram provided by the present application, as Figure 5 shown, assuming that there are two implementation versions of the operator, namely version A and version B, the test output of version A includes test tensors A1, A2, …, An, and the test output of version B includes test tensors B1, B2, …, Bn, wherein A1 and B1, A2 and B2, …, An and Bn are the performances of the same operator's test tensor in different versions. The Kappa coefficient of each test tensor is the inconsistency degree of each test tensor. When comparing versions A and B, the Kappa value of A1 can be compared with the Kappa value of B1, the Kappa value of A2 can be compared with the Kappa value of B2, and the Kappa value of An can be compared with the Kappa value of Bn, thereby comparing the inconsistency degrees of versions A and B.

[0112] Based on any of the above embodiments, the comparing the versions based on the degree of inconsistency of the operator in the multiple versions comprises:

[0113] based on the degree of inconsistency of each test tensor of the operator under multiple versions and the importance of each test tensor, performing version comparison.

[0114] Specifically, for the case where there are multiple test tensors for an operator, the degree of inconsistency of different test tensors under different versions may be different, for example, assuming that there are two versions A and B of the operator, among the 5 test tensors, the degree of inconsistency of 3 test tensors is that version A is better than version B, and the degree of inconsistency of the other 2 test tensors is that version B is better than version A, which makes it difficult to compare the overall degree of inconsistency of versions A and B.

[0115] For this case, the importance of the test tensor is introduced in the embodiments of the present application. That is, the importance of different test tensors in the operation of the operator, or even the importance of the operator in the artificial intelligence model, may be different. The importance of the test tensor will affect the judgment of the degree of inconsistency of the version. Therefore, when comparing the degree of inconsistency of the test tensor in different versions, the importance of the test tensor can be introduced. For example, the importance of the test tensor can be used as a weight, the degree of inconsistency of each test tensor under a version is weighted and summed, and the result obtained by the weighted summation is compared with the result of the weighted summation of another version, so as to judge which version has a higher or lower degree of inconsistency.

[0116] Here, the importance of the test tensor can be pre-set, or can be calculated based on the influence of the test tensor on the output of the model in the artificial intelligence model, and the embodiments of the present application do not make specific limitation.

[0117] based on any of the above embodiments, Figure 6 is a flowchart of the non-inconsistency degree evaluation method of the operator provided by the present application, as shown in Figure 6 , the method comprises:

[0118] First, before each execution of the operator, a copy operation is performed on the input tensor of the operator (corresponding to step ① in Figure 6 ), so as to ensure that the input tensor is the same each time the operator is executed. It should be noted that if the operator does not modify the input tensor, the above operation of copying the input tensor can be omitted. As an example, the input tensor can include tensor 0, tensor 1 and tensor 2.

[0119] Then, the operator is executed (corresponding to step ② in Figure 6 ). In this process, the same operator is executed N times, which corresponds to the first to Nth times in Figure 6 .

[0120] Then, the output obtained by the acquisition operator is obtained, i.e., the test tensors (corresponding to Figure 6 Step ③ in the foregoing method) are obtained. For a multi-output operator, all the output tensors of the operator need to be obtained, for example, tensor 3 and tensor 4 in Figure 6 . In addition, if the operator is executed for the first time, the test tensors are additionally saved as the benchmark tensors.

[0121] Subsequently, for each test tensor output by each execution of the operator, for each element on the test tensor, if the element is the same as the value of the corresponding element in the benchmark tensor, it is considered that the element is completely consistent with the benchmark tensor, and the score value of the corresponding element in the inconsistency score of the test tensor is updated to 1, otherwise to 0. After all the elements in a certain test tensor are updated in turn, all the test tensors are updated in turn (corresponding to Step ④ in the foregoing method). As an example, score 0 can be the inconsistency score of tensor 3, and score 1 can be the inconsistency score of tensor 4, and the score 0 and the score 1 of each execution of the operator can be obtained. Figure 6

[0122] Then, the next round of loop operation is started (corresponding to Step ⑤ in the foregoing method), until the Nth loop is executed. Figure 6

[0123] Finally, the inconsistency scores of the test tensors obtained by each execution are counted, and the kappa values of each test tensor are obtained by using the Kappa formula as the inconsistency degree of the operator (corresponding to Step ⑥ in the foregoing method). As an example, kappa 0 and kappa 1 are the inconsistency degrees of tensor 3 and tensor 4 respectively, and the two can represent the inconsistency degree of the operator. Figure 6

[0124] The inconsistency degree evaluation device of the operator provided by the present application is described below, and the inconsistency degree evaluation device of the operator described below can be correspondingly referred to the inconsistency degree evaluation method of the operator described above.

[0125] Figure 7 is a structural schematic diagram of the inconsistency degree evaluation device of the operator provided by the present application, as shown in Figure 7 , the device comprises:

[0126] The test unit 710 is configured to obtain the benchmark output of the operator and a plurality of groups of test outputs obtained by a plurality of executions of the operator.

[0127] ​​​The scoring unit 720 is configured to compare each group of test output with the reference output respectively by taking a tensor as a comparison unit, to obtain a non-consistency score of each test tensor in each group of test output compared with a reference tensor in the reference output, and the non-consistency score indicates whether each element in the test tensor and the reference tensor is consistent.

[0128] The degree evaluation unit 730 is configured to evaluate the non-consistency degree of the operator based on the non-consistency score of each test tensor in each group of test output.

[0129] In the device provided by the embodiments of the present application, the non-consistency score of each test tensor in the test output can be obtained by comparing each group of test output with the reference output, so as to reflect whether each element in the test tensor and the reference tensor is consistent, and the non-consistency degree of the operator is evaluated based on this, thereby realizing the quantification of the non-consistency of the operator, providing a basis for comparing the non-consistency degrees of different versions of the operator, and helping to accurately locate the stability problem of the artificial intelligence model and improve the porting efficiency of the artificial intelligence model adapted to the chip.

[0130] Based on the above-mentioned embodiments, the scoring unit is specifically configured to:

[0131] Each group of test output is compared with the reference output respectively by taking a tensor as a comparison unit, to obtain an element comparison result of each test tensor in each group of test output compared with the reference tensor.

[0132] For each test tensor in each group of test output, based on the element comparison result, a comparison score of each element in the test tensor is determined, and based on the position and comparison score of each element in the test tensor, a non-consistency score of the test tensor is determined.

[0133] Based on any of the above-mentioned embodiments, the degree evaluation unit is specifically configured to:

[0134] Based on the non-consistency scores of the same test tensor in each group of test output, the non-consistency degree of the test tensor is evaluated.

[0135] Based on the non-consistency degree of each test tensor, the non-consistency degree of the operator is determined.

[0136] Based on any of the above-mentioned embodiments, the degree evaluation unit is specifically configured to:

[0137] Based on the number of elements of the test tensor, the number of running times of the operator, and the non-consistency score of the test tensor in each group of test output, an observation consistency probability and an expected consistency probability of the test tensor are determined.

[0138] determine a degree of inconsistency of the test tensor based on the observed consistency probability and the expected consistency probability of the test tensor.

[0139] According to any one of the above embodiments, the degree evaluation unit is specifically configured to:

[0140] count a number of times that each element in the test tensor belongs to each score type in the inconsistency score based on the inconsistency score of each test tensor in each group of test outputs;

[0141] determine the observed consistency probability and the expected consistency probability of the test tensor based on the number of times that each element belongs to each score type in the inconsistency score, and the number of elements and the number of runs.

[0142] According to any one of the above embodiments, the apparatus further comprises a version comparison unit configured to:

[0143] perform version comparison based on the degrees of inconsistency of the operator in the plurality of versions.

[0144] According to any one of the above embodiments, the version comparison unit is specifically configured to:

[0145] perform version comparison based on the degrees of inconsistency of each test tensor of the operator in the plurality of versions and the importance of each test tensor.

[0146] Figure 8 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 8 The electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 can communicate with each other through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute an inconsistency degree evaluation method of an operator, the method comprising:

[0147] obtaining a reference output of an operator and a plurality of groups of test outputs obtained by running the operator a plurality of times;

[0148] comparing each group of test outputs with the reference output respectively as a comparison unit to obtain an inconsistency score of each test tensor in each group of test outputs compared with a reference tensor in the reference output, the inconsistency score indicating whether each element in the test tensor and the reference tensor is consistent or not;

[0149] evaluating a degree of inconsistency of the operator based on the inconsistency score of each test tensor in each group of test outputs.

[0150] Furthermore, the logic instructions in the memory 830 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the parts that contribute to the related art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0151] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the non-consistency degree evaluation method of the operator provided by the above-mentioned methods, the method comprising:

[0152] obtaining a reference output of the operator and a plurality of test outputs obtained by running the operator multiple times;

[0153] comparing each test output with the reference output respectively as a comparison unit of a tensor to obtain a non-consistency score of each test tensor in each test output compared with a reference tensor in the reference output, the non-consistency score indicating whether each element in the test tensor and the reference tensor is consistent or not;

[0154] evaluating the non-consistency degree of the operator based on the non-consistency score of each test tensor in each test output.

[0155] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the non-consistency degree evaluation method of the operator provided by the above-mentioned methods, the method comprising:

[0156] obtaining a reference output of the operator and a plurality of test outputs obtained by running the operator multiple times;

[0157] comparing each test output with the reference output respectively as a comparison unit of a tensor to obtain a non-consistency score of each test tensor in each test output compared with a reference tensor in the reference output, the non-consistency score indicating whether each element in the test tensor and the reference tensor is consistent or not;

[0158] Based on the inconsistency score of each test tensor in each set of test outputs, the degree of inconsistency of the operator is evaluated.

[0159] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0160] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating the inconsistency of an operator, characterized in that, The operator has multiple versions, and the method includes: For each of the multiple versions, obtain the baseline output of the operator and multiple sets of test outputs obtained from multiple runs of the operator. The baseline output and the test output are obtained respectively during the process of repeatedly running the operator under the same conditions. The same conditions include that the performance parameters of the hardware device running the operator are the same and the input is the same. For each of the multiple versions, using tensors as the comparison unit, each set of test outputs is compared with the benchmark output to obtain the non-consistency score of each test tensor in each set of test outputs compared with the benchmark tensor in the benchmark output. The non-consistency score indicates whether the elements in the test tensor and the benchmark tensor are consistent. For each of the multiple versions, the degree of inconsistency of the operator is evaluated based on the inconsistency score of each test tensor in each set of test outputs; Version comparison is performed based on the degree of inconsistency of the operator across the multiple versions; The multiple versions are different implementations of the operator; The process of comparing each set of test outputs with the benchmark output, using tensors as the comparison unit, to obtain a non-consistency score for each test tensor in each set of test outputs compared to the benchmark tensor in the benchmark output, includes: Using tensors as the comparison unit, each set of test outputs is compared with the benchmark output to obtain the element-wise comparison result of each test tensor in each set of test outputs compared with the benchmark tensor; For each test tensor in each set of test outputs, a comparison score for each element in the test tensor is determined based on the element comparison results. Based on the position and comparison score of each element in the test tensor, a non-consistency score for the test tensor is determined.

2. The method for evaluating the inconsistency of operators according to claim 1, characterized in that, The evaluation of the operator's inconsistency level based on the inconsistency score of each test tensor in each set of test outputs includes: The degree of inconsistency of the test tensor is evaluated based on the inconsistency score of the same test tensor in each set of test outputs. The degree of inconsistency of the operator is determined based on the degree of inconsistency of each test tensor.

3. The method for evaluating the inconsistency of operators according to claim 2, characterized in that, The evaluation of the degree of inconsistency of the test tensor based on the inconsistency score of the same test tensor in each set of test outputs includes: Based on the number of elements in the test tensor, the number of times the operator is run, and the non-consistency score of the test tensor in each test output, determine the observed consistency probability and the expected consistency probability of the test tensor. The degree of inconsistency of the test tensor is determined based on the observed consistency probability and the expected consistency probability of the test tensor.

4. The method for evaluating the inconsistency of operators according to claim 3, characterized in that, The determination of the observed consistency probability and expected consistency probability of the test tensor based on the number of elements in the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each test output includes: Based on the non-consistency score of the test tensor in each test output, the number of times each element in the test tensor belongs to each score type in the non-consistency score is counted. Based on the number of times each element belongs to each score type in the inconsistency score, as well as the number of elements and the number of runs, the observed consistency probability and expected consistency probability of the test tensor are determined.

5. The method for evaluating the inconsistency of operators according to claim 1, characterized in that, The version comparison based on the degree of inconsistency of the operator across multiple versions includes: Version comparison is performed based on the degree of inconsistency of the operator in each test tensor across the multiple versions, and the importance of each test tensor.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for evaluating the degree of inconsistency of the operator as described in any one of claims 1 to 5.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for evaluating the degree of inconsistency of the operator as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for evaluating the degree of inconsistency of the operator as described in any one of claims 1 to 5.

Images