Non-consistency degree evaluation method of operator, electronic equipment and storage medium

Abstract

The invention provides an operator non-consistency degree evaluation method, electronic equipment and a storage medium. The method comprises the following steps: acquiring a reference output of an operator and multiple groups of test outputs obtained by multiple times of operation of the operator; the tensor is used as a comparison unit, each group of test output is compared with the reference output, the non-consistency score of each test tensor in each group of test output compared with the reference tensor in the reference output is obtained, and the non-consistency score represents whether elements in the test tensor and the reference tensor are consistent or not; and evaluating the non-consistency degree of the operator based on the non-consistency score of each test tensor in each group of test outputs. According to the method, the electronic equipment and the storage medium provided by the invention, the quantification of the non-consistency of the operator is realized, so that a basis is provided for comparing the non-consistency degrees of different versions of the operator, the stability problem of the artificial intelligence model can be accurately positioned, and the transplantation efficiency of an adaptive chip of the artificial intelligence model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator inconsistency degree evaluation method, electronic equipment and storage medium. Background Art

[0002] Consistency evaluation is an important means of measuring the stability of operators. It is implemented by applying fixed input to the operator and executing the operator multiple times to determine whether its output is exactly the same. If during multiple executions, at least one of the outputs is inconsistent with the outputs of other rounds, the operator is judged as an inconsistent operator.

[0003] However, the current consistency evaluation method for operators has obvious limitations. Specifically, it can only qualitatively determine whether an operator is consistent, but cannot further quantify the degree of operator inconsistency. In fact, the degree of inconsistency will directly affect the size of random fluctuations caused by the application of operators in artificial intelligence models, thereby affecting the accuracy and stability of artificial intelligence models.

[0004] Therefore, how to quantify the degree of operator inconsistency remains a problem that needs to be solved urgently in this field. Summary of the invention

[0005] The present invention provides an operator inconsistency degree evaluation method, an electronic device and a storage medium, which are used to solve the defect that the related art only performs qualitative analysis on whether an operator has consistency and cannot meet the actual application requirements.

[0006] The present invention provides a method for evaluating the inconsistency degree of an operator, comprising: Obtaining a benchmark output of an operator and multiple sets of test outputs obtained by running the operator multiple times; Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; Based on the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator is evaluated.

[0007] According to a method for evaluating the inconsistency degree of an operator provided by the present invention, taking tensors as comparison units, comparing each group of test outputs with the benchmark outputs respectively, and obtaining an inconsistency score of each test tensor in each group of test outputs compared with a benchmark tensor in the benchmark output, comprises: Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain element comparison results of each test tensor in each group of test outputs compared with the benchmark tensor; For each test tensor in each set of test outputs, a comparison score for each element in the test tensor is determined based on the element comparison result, and an inconsistency score for the test tensor is determined based on the position and comparison score of each element in the test tensor.

[0008] According to a method for evaluating the inconsistency degree of an operator provided by the present invention, the inconsistency degree of the operator is evaluated based on the inconsistency score of each test tensor in each group of test outputs, including: Based on the inconsistency score of the same test tensor in each group of test outputs, evaluating the inconsistency degree of the test tensor; Based on the inconsistency degree of each test tensor, the inconsistency degree of the operator is determined.

[0009] According to a method for evaluating the inconsistency degree of an operator provided by the present invention, the inconsistency degree of the test tensor is evaluated based on the inconsistency score of the same test tensor in each group of test outputs, including: Determine an observed consistency probability and an expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each set of test outputs; Based on the observed consistency probability and the expected consistency probability of the test tensor, the degree of inconsistency of the test tensor is determined.

[0010] According to a method for evaluating the inconsistency degree of an operator provided by the present invention, the method determines the observed consistency probability and the expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of operations of the operator, and the inconsistency score of the test tensor in each group of test outputs, including: Based on the inconsistency score of the test tensor in each set of test outputs, counting the number of times each element in the test tensor belongs to various score types in the inconsistency score; Based on the number of times each element belongs to various score types in the inconsistency score, as well as the number of elements and the number of runs, an observed consistency probability and an expected consistency probability of the test tensor are determined.

[0011] A method for evaluating the inconsistency degree of an operator provided by the present invention further includes: Based on the inconsistency degree of the operator under multiple versions, version comparison is performed.

[0012] According to a method for evaluating the inconsistency degree of an operator provided by the present invention, performing version comparison based on the inconsistency degree of the operator under multiple versions includes: Version comparison is performed based on the inconsistency degree of each test tensor of the operator under multiple versions and the importance of each test tensor.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for evaluating the degree of inconsistency of the operator described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for evaluating the degree of inconsistency of an operator as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for evaluating the degree of inconsistency of an operator as described above is implemented.

[0016] The operator inconsistency degree evaluation method, electronic device and storage medium provided by the present invention can obtain the inconsistency score of each test tensor in the test output by comparing each group of test outputs and benchmark outputs, thereby reflecting whether the elements in the test tensor and the benchmark tensor are consistent, and evaluate the inconsistency degree of the operator on this basis, thereby realizing quantification of operator inconsistency, thereby providing a basis for comparing the inconsistency degrees of different versions of the operator, helping to accurately locate the stability problems of the artificial intelligence model and improve the transplantation efficiency of the artificial intelligence model adaptation chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is one of the flow charts of the method for evaluating the degree of inconsistency of an operator provided by the present invention; Figure 2 It is one of the comparison schematic diagrams of the test tensor and the benchmark tensor provided by the present invention; Figure 3 This is the second schematic diagram of comparison between the test tensor and the benchmark tensor provided by the present invention; Figure 4This is the third schematic diagram of comparison between the test tensor and the benchmark tensor provided by the present invention; Figure 5 It is a schematic diagram of version comparison provided by the present invention; Figure 6 This is the second flow chart of the method for evaluating the inconsistency degree of an operator provided by the present invention; Figure 7 It is a structural schematic diagram of the inconsistency degree evaluation device of the operator provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] In the field of artificial intelligence technology, the evaluation of operators is a crucial link, which is directly related to the performance and accuracy of artificial intelligence models. Usually, when evaluating an operator, a comprehensive consideration is made from multiple dimensions, including but not limited to key indicators such as performance, accuracy, and consistency.

[0021] Among them, consistency evaluation is an important means to measure the stability of operators. Its implementation method is mainly to apply fixed input to the operator and execute the operator multiple times to determine whether its output is exactly the same. If during multiple executions, at least one output is inconsistent with the output of other rounds, the operator is judged as an inconsistent operator.

[0022] However, the current consistency evaluation method for operators has obvious limitations. Specifically, it can only qualitatively determine whether an operator is consistent. In other words, the current consistency evaluation of operators is only used to determine whether an operator is consistent. It is a qualitative evaluation of the operator. This evaluation indicator is too simple and lacks quantitative evaluation. It is even more impossible to evaluate or compare the advantages and disadvantages of multiple versions when there are multiple versions of the operator. It can be understood that the multiple versions of the operator referred to here refer to different implementation methods of the operator, which can be specifically reflected in different segmentation dimensions, different segmentation granularity, etc.

[0023] Specifically, this qualitative evaluation method is inadequate when faced with complex and ever-changing AI application scenarios. In particular, in AI models, although some inconsistency operators may be similar to other operators in performance, the random fluctuation values ​​of their outputs may have a serious impact on the overall stability of the model. Here, random fluctuation refers to the situation where the output of an inconsistency operator is inconsistent and fluctuates during multiple executions of the same input. The random fluctuation value is the specific difference data caused by random fluctuations.

[0024] For example, for two implementation versions A and B of an operator, if version A is comparable to version B in performance, but the output of version A contains more random fluctuations than the output of version B, this fluctuation is likely to cause great adverse interference to the artificial intelligence model. However, since the current consistency evaluation of operators is mostly qualitative, assuming that both version A and version B are evaluated as inconsistent operators, researchers cannot judge whether to apply version A or version B in the artificial intelligence model only through qualitative consistency evaluation. The fact that the output of version A contains more random fluctuations than the output of version B can actually be reflected through quantitative consistency evaluation methods, that is, if the degree of inconsistency of the operator can be quantified, then the quantified degree of inconsistency can provide an effective reference for whether to apply version A or version B in the artificial intelligence model.

[0025] Therefore, how to quantify the degree of operator inconsistency remains a problem that needs to be solved urgently in this field.

[0026] To solve this problem, the present invention provides an operator inconsistency degree evaluation method. Figure 1 is one of the flow charts of the method for evaluating the inconsistency degree of an operator provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, obtaining the benchmark output of the operator and multiple groups of test outputs obtained by running the operator multiple times.

[0027] The operator here is the operator that needs to be evaluated for the degree of inconsistency.

[0028] The benchmark output and test output of an operator are both the outputs obtained from the operation of the operator, and the benchmark output and test output are obtained respectively when the operator is repeatedly run multiple times under the same conditions. The same conditions here include the same operator operation environment, that is, the same hardware equipment running the operator, or the same performance parameters of the hardware equipment running the operator, and the same input when the operator is running.

[0029] For example, you can repeatedly run the operator multiple times under the same conditions, use the first output as the benchmark output, and use the subsequent outputs as the test outputs; or, you can repeatedly run the operator multiple times under the same conditions, use the output obtained at any one time as the benchmark output, and use the outputs obtained at the remaining times as the test outputs.

[0030] It can be understood that the benchmark output is the output used as the benchmark (Golden) for output comparison when evaluating the inconsistency of the operator, and the test output is the output compared with the benchmark output when evaluating the inconsistency of the operator. After obtaining the benchmark output and test output of the operator, the inconsistency of the operator can be evaluated based on them.

[0031] Step 120, taking tensors as comparison units, compare each group of test outputs with the benchmark outputs respectively, and obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor.

[0032] Specifically, the output of the operator may include one or more tensors. In the test output, the above tensors are recorded as test tensors, and in the benchmark output, the above tensors are recorded as benchmark tensors. It is understood that the number and type of test tensors in the test output are the same as the number and type of benchmark tensors in the benchmark output, that is, the test tensors in the test output correspond one-to-one to the benchmark tensors in the benchmark output.

[0033] Therefore, for each set of test outputs, tensors can be used as comparison units to compare each test tensor in the test outputs with the corresponding benchmark tensor in the benchmark outputs.

[0034] In an embodiment of the present invention, the process of performing a one-to-one comparison between the test tensor and the reference tensor can be regarded as a score for scoring the inconsistency between the test tensor and the reference tensor, and thus, the result of the comparison between the test tensor and the reference tensor is used as the inconsistency score of the test tensor compared to the reference tensor. Here, for each set of test outputs, each test tensor in the test output can obtain a corresponding inconsistency score, and the inconsistency score here can reflect whether each element in the test tensor is consistent with the element at the corresponding position in the reference tensor.

[0035] Specifically, for a test tensor and its corresponding reference tensor, the information that can reflect the inconsistency between the two includes the number of inconsistent elements in the test tensor and the reference tensor, the positions of the inconsistent elements in the test tensor and the reference tensor, and the degree of inconsistency between the inconsistent elements in the test tensor and the reference tensor.

[0036] For example, Figure 2 is one of the comparison diagrams of the test tensor and the benchmark tensor provided by the present invention, such as Figure 2 As shown, assuming that the benchmark tensor and test tensors 1 and 2 are all 3×3 matrices, by comparing the benchmark tensor with test tensor 1, it can be seen that the elements of test tensor 1 are inconsistent with those of the benchmark tensor at the two coordinate positions (2,1) and (2,2); by comparing the benchmark tensor with test tensor 2, it can be seen that the elements of test tensor 2 are inconsistent with those of the benchmark tensor at the two coordinate positions (1,1) and (3,3). It can be seen that the positions of inconsistent elements of different test tensors may be different compared with the benchmark tensor.

[0037] For example, Figure 3 This is the second schematic diagram of the comparison between the test tensor and the benchmark tensor provided by the present invention. Figure 3 As shown, assuming that the benchmark tensor and test tensors 1 and 2 are all 3×3 matrices, by comparing the benchmark tensor with test tensor 1, it can be seen that test tensor 1 has one element inconsistent with the benchmark tensor; by comparing the benchmark tensor with test tensor 2, it can be seen that test tensor 2 has two elements inconsistent with the benchmark tensor. It can be seen that different test tensors may have different numbers of inconsistent elements compared with the benchmark tensor.

[0038] For example, Figure 4 3 is a schematic diagram of comparison between the test tensor and the reference tensor provided by the present invention. Figure 4 As shown, assuming that the benchmark tensor and test tensor 1 and test tensor 2 are all 3×3 matrices, by comparing the benchmark tensor with test tensor 1, it can be known that the value of the elements of test tensor 1 that are inconsistent with the benchmark tensor is 9; by comparing the benchmark tensor with test tensor 2, it can be known that the value of the elements of test tensor 2 that are inconsistent with the benchmark tensor is 5.2. It can be seen that different test tensors may have different values ​​of inconsistent elements compared with the benchmark tensor, and the magnitude of the inconsistent value fluctuations reflected may also be different.

[0039] The inconsistency score reflecting whether each element in the test tensor is consistent with the element at the corresponding position in the reference tensor can cover the degree of inconsistency reflected by at least one of the position, number and value of the inconsistent elements. For example, the inconsistency score can be expressed as a tensor obtained by subtracting the test tensor from the reference tensor, or the inconsistency score can be expressed as a tensor of the same size as the test tensor and the reference tensor, in which the value of each element is used to reflect whether the elements at the corresponding positions of the test tensor and the reference tensor are consistent. The embodiments of the present invention do not specifically limit this.

[0040] Step 130 , based on the inconsistency score of each test tensor in each set of test outputs, evaluate the inconsistency degree of the operator.

[0041] Specifically, after obtaining the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator can be evaluated based on the score. Here, when evaluating the inconsistency degree of the operator, considering that the inconsistency degrees of different output tensors may be different, the inconsistency degree of the operator can be specifically expressed as the inconsistency degree of each tensor in the operator output.

[0042] Thus, for any tensor, the inconsistency score of the tensor in each set of test outputs can be counted, and the degree of inconsistency of the tensor can be calculated based on this. For example, the kappa coefficient of the inconsistency score of the tensor in each set of test outputs can be calculated as the degree of inconsistency of the tensor; for another example, the Krippendorff's Alpha parameter of the inconsistency score of the tensor in each set of test outputs can be calculated as the degree of inconsistency of the tensor, and the embodiment of the present invention does not specifically limit this. It can be understood that the kappa coefficient and Krippendorff's Alpha parameter here are both indicators for consistency verification.

[0043] In the method provided in an embodiment of the present invention, by comparing each group of test outputs and benchmark outputs, the inconsistency score of each test tensor in the test output can be obtained, thereby reflecting whether the elements in the test tensor and the benchmark tensor are consistent, and on this basis, the degree of inconsistency of the operator is evaluated, thereby achieving quantification of operator inconsistency, thereby providing a basis for comparing the degree of inconsistency of different versions of the operator, helping to accurately locate the stability problems of the artificial intelligence model and improve the transplantation efficiency of the artificial intelligence model adaptation chip.

[0044] Based on the above embodiment, in step 120, taking tensors as comparison units, comparing each group of test outputs with the benchmark outputs, respectively, to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, includes: Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain element comparison results of each test tensor in each group of test outputs compared with the benchmark tensor; For each test tensor in each set of test outputs, a comparison score for each element in the test tensor is determined based on the element comparison result, and an inconsistency score for the test tensor is determined based on the position and comparison score of each element in the test tensor.

[0045] Specifically, in the comparison process using tensors as comparison units, for each test tensor in each set of test outputs, an element comparison result of the test tensor compared to the corresponding reference tensor can be obtained. Here, the element comparison result may include a comparison result of each element in the test tensor compared to a corresponding element in the reference tensor, specifically, whether each element in the test tensor is consistent with a corresponding element in the reference tensor.

[0046] For each test tensor, a comparison score can be assigned to each element in the test tensor based on the element comparison result. For example, for an element judged to be consistent in the element comparison result, the comparison score of the element can be assigned to 1, and for an element judged to be inconsistent in the element comparison result, the comparison score of the element can be assigned to 0.

[0047] After obtaining the comparison score of each element in the test tensor, the comparison score of each element and the position of each element in the test tensor can be combined to construct an inconsistency score that can characterize whether the elements in the test tensor and the benchmark tensor are consistent.

[0048] Here, the inconsistency score can be in the form of a tensor, the size of the inconsistency score can be consistent with the size of the test tensor, and the element value of the inconsistency score is the comparison score of the element at the corresponding position in the test tensor. Figure 2 The comparison scores of the test tensor 1 at positions (2,1) and (2,2) are 0, and the comparison scores at other positions are 1. Thus, the inconsistency scores can be obtained in the following form: Alternatively, the inconsistency score can also be in the form of a list, which is specifically a mapping table between the coordinate value of each element in the test tensor and the comparison score of the element. The comparison score of the element at the corresponding position of the coordinate value can be obtained by looking up the coordinate value. Figure 2 The comparison scores of the test tensor 1 at positions (2,1) and (2,2) are 0, and the comparison scores at other positions are 1. Figure 2 For the test tensor 2 in , the comparison score of the test tensor 2 at the two positions (1,1) and (3,3) is 0, and the comparison score of the other positions is 1. Therefore, the inconsistency scores in the following table can be obtained for the test tensors 1 and 2:

[0049] Based on any of the above embodiments, in step 130, the step of evaluating the inconsistency degree of the operator based on the inconsistency score of each test tensor in each group of test outputs includes: Based on the inconsistency score of the same test tensor in each group of test outputs, evaluating the inconsistency degree of the test tensor; Based on the inconsistency degree of each test tensor, the inconsistency degree of the operator is determined.

[0050] Specifically, considering that the inconsistency degree of different output tensors may be different, the inconsistency degree of an operator can be specifically expressed as the inconsistency degree of each tensor in the operator output.

[0051] Therefore, for any tensor, the inconsistency score of the tensor in each set of test outputs can be counted, and the inconsistency degree of the tensor can be calculated based on this.

[0052] For example, for a test tensor, the inconsistent score of the test tensor in each set of test outputs can be used as an evaluation object, and the Kappa coefficient between the above evaluation objects can be calculated to measure the consistency between the evaluation objects. The calculated Kappa coefficient can be used as the inconsistency degree of the test tensor.

[0053] For another example, for a test tensor, the Krippendorff's Alpha parameter can be applied as the inconsistency degree of the test tensor. Specifically, the Krippendorff's Alpha parameter is widely used in the fields of content analysis, text encoding, survey data encoding, etc., and is used to measure the consistency degree when different coders encode the same set of data. Based on the above ideas, in the process of using the Krippendorff's Alpha parameter as the inconsistency degree of the test tensor, each element in the test tensor in a set of test outputs can be regarded as a text fragment, each set of test outputs can be regarded as an encoder, and whether the elements are consistent can be regarded as the encoding result, that is, each operation of the operator can be understood as an experiment, so that an encoding result can be generated in each experiment, that is, the inconsistency score of the test tensor in the test output obtained when the operator is run each time. Therefore, the inconsistency score of the test tensor in each set of test outputs can be regarded as the encoding result generated in each experiment to calculate the Krippendorff's Alpha parameter, and the calculated Krippendorff's Alpha parameter is used as the inconsistency degree of the test tensor.

[0054] After obtaining the inconsistency degree of each test tensor, the inconsistency degree of each test tensor can be summarized as the inconsistency degree of the operator.

[0055] Based on any of the above embodiments, in step 130, the step of evaluating the inconsistency degree of the test tensor based on the inconsistency score of the same test tensor in each group of test outputs includes: Determine an observed consistency probability and an expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each set of test outputs; Based on the observed consistency probability and the expected consistency probability of the test tensor, the degree of inconsistency of the test tensor is determined.

[0056] Specifically, when evaluating the inconsistency degree for a test tensor, it can be done by calculating the Kappa coefficient. Here, the calculation of the Kappa coefficient can be implemented based on the observed consistency probability and the expected consistency probability.

[0057] Among them, the probability of observation agreement is usually expressed as , the observed consistency probability refers to the consistency degree observed for the evaluation object when the inconsistency score of the test tensor in each group of test outputs is used as the evaluation object. The observed consistency probability reflects the closeness of the inconsistency scores in each group of test outputs in the classification score of whether the elements are consistent. The observed consistency probability can be calculated by the number of elements in the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each group of test outputs. Here, the number of times the operator is run is the number of test output groups.

[0058] The expected probability of agreement is usually expressed as , the expected consistency probability refers to the probability that the inconsistency scores of the test tensor in each set of test outputs are consistent due to randomness, without considering the inconsistency scores of the test tensor in each set of test outputs. The expected consistency probability can reflect the possibility that the inconsistency scores in each set of test outputs are consistent when the elements are classified and scored completely randomly based on whether they are consistent. The expected consistency probability can be calculated by the number of elements in the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each set of test outputs. Here, the number of times the operator is run is the number of test output groups.

[0059] After obtaining the observed consistency probability and the expected consistency probability, the inconsistency degree of the test tensor can be calculated, for example, it can be expressed as the following formula: In the formula, To test the degree of inconsistency of the tensor.

[0060] Based on any of the above embodiments, in step 130, determining the observed consistency probability and the expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of operations of the operator, and the inconsistency score of the test tensor in each group of test outputs includes: Based on the inconsistency score of the test tensor in each set of test outputs, counting the number of times each element in the test tensor belongs to various score types in the inconsistency score; Based on the number of times each element belongs to various score types in the inconsistency score, as well as the number of elements and the number of runs, an observed consistency probability and an expected consistency probability of the test tensor are determined.

[0061] Specifically, the inconsistency score of the test tensor in any set of test outputs may include the score type of each element in the test tensor in the set of test outputs. For example, if the score is 1, it indicates consistency, and if the score is 0, it indicates inconsistency, then the score type may include 1 and 0. Thus, after obtaining the inconsistency score of the test tensor in each set of test outputs, the score type of each element in the test tensor in each set of test outputs may be counted, thereby obtaining the number of times each element belongs to various score types in the inconsistency score. For example, in a total of 1,000 sets of test outputs, an element belongs to score type 1 900 times and belongs to score type 0 100 times.

[0062] From there, we can calculate observed and expected agreement probabilities for the test tensor based on the number of times each element belongs to each score type in the inconsistency score, as well as the number of elements and number of runs mentioned above.

[0063] Among them, the calculation method for the probability of observation consistency can be expressed as follows: In the formula, is the number of times the operator is run, are indices into the test tensor, is the index of the score type in the inconsistency score, is the total number of categories scored for an element, . For the The number of times an element is classified into the jth category, for example For the The number of times the score type of an element is 1, For the The number of times the score type of an element is 0. For the The observed consistent probability of an element.

[0064] On this basis, the observation consistency probability of the test tensor can be calculated by the following formula: In the formula, is the observed consistent probability of the test tensor, is the total number of elements contained in the test tensor, The value of depends on the specific dimensions of the test tensor. For example, if the shape of the test tensor is [batch, height, width], then .

[0065] The calculation method for the expected consistency probability can be expressed as follows: In the formula, is the expected consistent probability of the jth score type. is the expected agreement probability of the test tensor.

[0066] Based on any of the above embodiments, the method further includes: Based on the inconsistency degree of the operator under multiple versions, version comparison is performed.

[0067] Specifically, when an operator has multiple implementation versions, each version of the operator can be treated as an independent operator, and the above-mentioned inconsistency evaluation method is performed on each version of the operator, thereby obtaining the inconsistency degree of each version of the operator, that is, the inconsistency degree of the operator in each version.

[0068] In this case, the different versions of the same operator can be compared by comparing the degree of inconsistency of the operator in each version, thereby providing conditions for selecting the operator version used in the artificial intelligence model.

[0069] Here, for the comparison of different versions of the operator, the inconsistency degree of the same test tensor under different versions can be compared. For example, Figure 5 is a schematic diagram of version comparison provided by the present invention, such as Figure 5 As shown, it is assumed that there are two implementation versions for the operator, namely version A and version B. The test output of version A includes test tensors A1, A2, ..., An, and the test output of version B includes test tensors B1, B2, ..., Bn, where A1 and B1, A2 and B2, ..., An and Bn are the performances of the test tensors of the same operator in different versions. The Kappa coefficient of each test tensor is the degree of inconsistency of each test tensor. When comparing versions A and B, the Kappa value of A1 can be compared with the Kappa value of B1, the Kappa value of A2 can be compared with the Kappa value of B2, and the Kappa value of An can be compared with the Kappa value of Bn, so as to compare the degree of inconsistency between version A and version B.

[0070] Based on any of the foregoing embodiments, performing version comparison based on the inconsistency degree of the operator under multiple versions includes: Version comparison is performed based on the inconsistency degree of each test tensor of the operator under multiple versions and the importance of each test tensor.

[0071] Specifically, for the case where there are multiple test tensors for an operator, considering that the inconsistency degrees of different test tensors in different versions may have different advantages and disadvantages, for example, assuming that there are two versions of the operator, A and B, among the five test tensors, the inconsistency degree of three test tensors shows that version A is better than version B, and the inconsistency degree of the other two test tensors shows that version B is better than version A, which makes it difficult to make an overall comparative evaluation of the inconsistency degree of versions A and B.

[0072] In view of this situation, the importance of the test tensor is introduced in an embodiment of the present invention. That is, the importance of different test tensors in the operation of the operator and even the importance of the operator in the artificial intelligence model may be different. The importance of the test tensor will affect the judgment of the degree of inconsistency of the version. Therefore, when comparing the degree of inconsistency of the test tensors in different versions, the importance of the test tensor can be introduced. For example, the importance of the test tensor can be used as a weight to perform a weighted summation on the inconsistency degree of each test tensor under one version, and the result of the weighted summation is compared with the result of the weighted summation of another version, thereby judging which of the two versions has a higher or lower degree of inconsistency.

[0073] Here, the importance of the test tensor can be pre-set or calculated based on the impact of the test tensor on the model output in the artificial intelligence model. The embodiment of the present invention does not specifically limit this.

[0074] Based on any of the above embodiments, Figure 6 This is the second flow chart of the method for evaluating the inconsistency degree of an operator provided by the present invention. Figure 6 As shown, the method includes: First, before each execution of the operator, a copy operation is performed on the input tensor of the operator (corresponding to Figure 6 Step ① in ), thereby ensuring that the input tensor is the same each time the operator is executed. It should be noted that if the operator does not modify the input tensor, the above operation of copying the input tensor can also be omitted. As an example, the input tensor may include tensor 0, tensor 1, and tensor 2.

[0075] Then, start executing the operator (corresponding to Figure 6 In this process, the same operator will be executed N times, corresponding to Figure 6 The first to Nth time in .

[0076] Next, get the output of the operator execution, that is, get the test tensor (corresponding to Figure 6 In step ③), for multi-output operators, you need to obtain the tensors of all their outputs, such as Figure 6 In addition, if the operator is executed for the first time, these test tensors are saved as benchmark tensors.

[0077] Then, for each test tensor output by the operator each time, for each element on the test tensor, if its value is the same as the corresponding element in the base tensor, the element is considered to be completely consistent with the base tensor, and the score value of the corresponding element in the inconsistency score of the test tensor is updated to 1, otherwise it is 0. After updating all elements in a test tensor in turn, and then updating all test tensors (corresponding Figure 6 As an example, score 0 may be the inconsistency score of tensor 3, score 1 may be the inconsistency score of tensor 4, and each time the operator is executed, score 0 and score 1 of the execution can be obtained.

[0078] Then, the next cycle operation begins (corresponding to Figure 6 Repeat step ⑤ in the above code until N cycles are completed.

[0079] Finally, the inconsistency score of the test tensor obtained from each execution is counted, and the Kappa formula is used to obtain the kappa value of each test tensor as the inconsistency degree of the operator (corresponding to Figure 6 As an example, kappa0 and kappa1 are the inconsistency levels of tensor 3 and tensor 4, respectively, which can represent the inconsistency level of the operator.

[0080] The following is a description of an operator inconsistency degree evaluation device provided by the present invention. The operator inconsistency degree evaluation device described below and the operator inconsistency degree evaluation method described above can be referred to each other.

[0081] Figure 7 is a schematic diagram of the structure of the inconsistency degree evaluation device of the operator provided by the present invention, such as Figure 7 As shown, the device comprises: A test unit 710, used to obtain a benchmark output of an operator and multiple groups of test outputs obtained by running the operator multiple times; A scoring unit 720 is used to compare each group of test outputs with the benchmark outputs using tensors as comparison units, and obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; The degree evaluation unit 730 is used to evaluate the inconsistency degree of the operator based on the inconsistency score of each test tensor in each set of test outputs.

[0082] In the device provided by the embodiment of the present invention, by comparing each group of test outputs and benchmark outputs, the inconsistency score of each test tensor in the test output can be obtained, thereby reflecting whether the elements in the test tensor and the benchmark tensor are consistent, and on this basis, the degree of inconsistency of the operator is evaluated, thereby achieving quantification of the operator inconsistency, thereby providing a basis for comparing the inconsistency degrees of different versions of the operator, helping to accurately locate the stability problems of the artificial intelligence model and improve the transplantation efficiency of the artificial intelligence model adaptation chip.

[0083] Based on the above embodiment, the scoring unit is specifically used for: Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain element comparison results of each test tensor in each group of test outputs compared with the benchmark tensor; For each test tensor in each set of test outputs, a comparison score for each element in the test tensor is determined based on the element comparison result, and an inconsistency score for the test tensor is determined based on the position and comparison score of each element in the test tensor.

[0084] Based on any of the above embodiments, the degree assessment unit is specifically used for: Based on the inconsistency score of the same test tensor in each group of test outputs, evaluating the inconsistency degree of the test tensor; Based on the inconsistency degree of each test tensor, the inconsistency degree of the operator is determined.

[0085] Based on any of the above embodiments, the degree assessment unit is specifically used for: Determine an observed consistency probability and an expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each set of test outputs; Based on the observed consistency probability and the expected consistency probability of the test tensor, the degree of inconsistency of the test tensor is determined.

[0086] Based on any of the above embodiments, the degree assessment unit is specifically used for: Based on the inconsistency score of the test tensor in each set of test outputs, counting the number of times each element in the test tensor belongs to various score types in the inconsistency score; Based on the number of times each element belongs to various score types in the inconsistency score, as well as the number of elements and the number of runs, an observed consistency probability and an expected consistency probability of the test tensor are determined.

[0087] Based on any of the above embodiments, the device further includes a version comparison unit, configured to: Based on the inconsistency degree of the operator under multiple versions, version comparison is performed.

[0088] Based on any of the above embodiments, the version comparison unit is specifically used for: Version comparison is performed based on the inconsistency degree of each test tensor of the operator under multiple versions and the importance of each test tensor.

[0089] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the inconsistency degree evaluation method of the operator, and the method includes: Obtaining a benchmark output of an operator and multiple sets of test outputs obtained by running the operator multiple times; Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; Based on the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator is evaluated.

[0090] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0091] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the inconsistency degree evaluation method of the operator provided by the above methods, the method comprising: Obtaining a benchmark output of an operator and multiple sets of test outputs obtained by running the operator multiple times; Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; Based on the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator is evaluated.

[0092] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the inconsistency degree evaluation method of the operator provided by the above methods, the method comprising: Obtaining a benchmark output of an operator and multiple sets of test outputs obtained by running the operator multiple times; Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; Based on the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator is evaluated.

[0093] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the inconsistency degree of an operator, characterized in that: include: Obtaining a benchmark output of an operator and multiple sets of test outputs obtained by running the operator multiple times; Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain an inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark output, wherein the inconsistency score indicates whether each element in the test tensor is consistent with that in the benchmark tensor; Based on the inconsistency score of each test tensor in each set of test outputs, the inconsistency degree of the operator is evaluated.

2. The method for evaluating the inconsistency degree of an operator according to claim 1, characterized in that: The method of comparing each group of test outputs with the benchmark outputs by taking the tensor as the comparison unit to obtain the inconsistency score of each test tensor in each group of test outputs compared with the benchmark tensor in the benchmark outputs includes: Taking tensors as comparison units, each group of test outputs is compared with the benchmark outputs to obtain element comparison results of each test tensor in each group of test outputs compared with the benchmark tensor; For each test tensor in each set of test outputs, a comparison score for each element in the test tensor is determined based on the element comparison result, and an inconsistency score for the test tensor is determined based on the position and comparison score of each element in the test tensor.

3. The method for evaluating the inconsistency degree of an operator according to claim 1, characterized in that: The step of evaluating the inconsistency degree of the operator based on the inconsistency score of each test tensor in each set of test outputs includes: Based on the inconsistency score of the same test tensor in each group of test outputs, evaluating the inconsistency degree of the test tensor; Based on the inconsistency degree of each test tensor, the inconsistency degree of the operator is determined.

4. The method for evaluating the inconsistency degree of an operator according to claim 3, characterized in that: The step of evaluating the inconsistency degree of the test tensor based on the inconsistency score of the same test tensor in each group of test outputs includes: Determine an observed consistency probability and an expected consistency probability of the test tensor based on the number of elements of the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each set of test outputs; Based on the observed consistency probability and the expected consistency probability of the test tensor, the degree of inconsistency of the test tensor is determined.

5. The method for evaluating the inconsistency degree of an operator according to claim 4, characterized in that: The determining, based on the number of elements of the test tensor, the number of times the operator is run, and the inconsistency score of the test tensor in each group of test outputs, the observed consistency probability and the expected consistency probability of the test tensor comprises: Based on the inconsistency score of the test tensor in each set of test outputs, counting the number of times each element in the test tensor belongs to various score types in the inconsistency score; Based on the number of times each element belongs to various score types in the inconsistency score, as well as the number of elements and the number of runs, an observed consistency probability and an expected consistency probability of the test tensor are determined.

6. The method for evaluating the inconsistency degree of an operator according to any one of claims 1 to 5, characterized in that: Also includes: Based on the inconsistency degree of the operator under multiple versions, version comparison is performed.

7. The method for evaluating the inconsistency degree of an operator according to claim 6, characterized in that: The performing version comparison based on the inconsistency degree of the operator under multiple versions includes: Version comparison is performed based on the inconsistency degree of each test tensor of the operator under multiple versions and the importance of each test tensor.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for evaluating the degree of inconsistency of the operator according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for evaluating the degree of inconsistency of an operator according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for evaluating the degree of inconsistency of an operator according to any one of claims 1 to 6 is implemented.

Images