A model verification method, device, storage medium and product

Abstract

The embodiment of the application provides a model verification method, device, storage medium and product, relates to the technical field of artificial intelligence chips, and the method comprises the following steps: when a target model is migrated from a reference hardware platform to an artificial intelligence chip, a plurality of sub-sample sets extracted from an evaluation sample set are used, and a gradual verification strategy is used to perform verification step by step. In each verification process, the accuracy difference between the verification accuracy and the reference accuracy is calculated, and when the accuracy difference is greater than a preset accuracy threshold, the model verification is ended, and the verification failure is taken as a verification result, so that the accuracy problem existing in the target model can be quickly found and loss can be timely stopped, thereby effectively reducing resource waste and reducing model accuracy verification time. The end-to-end inference is performed on the artificial intelligence chip to obtain the verification accuracy of the model level task accuracy, and then the accuracy difference between the verification accuracy and the reference accuracy is obtained to obtain the verification result of the model level. Compared with the operator level accuracy verification method, the accuracy is higher.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a model verification method, device, storage medium and product. Background Technology

[0002] Artificial intelligence (AI) models typically refer to neural network models trained for inference and prediction, and AI chips are usually used to implement the computation of AI models. In practical applications, after migrating AI models from general-purpose chips to self-developed accelerator chips, it is necessary to verify whether the model's accuracy on the new hardware meets the requirements.

[0003] Under relevant technologies, the full-scale execution method is used to validate AI models, which involves using a complete evaluation sample set to perform large-scale benchmark validation on the AI ​​model and obtain validation results. However, the complete evaluation sample set has a large amount of data; for example, the full-scale Massive Multitask Language Understanding (MMLU) set includes 14,042 questions. Thus, using the full-scale execution method to validate AI models results in long validation times and high computational resource consumption. Summary of the Invention

[0004] This application provides a model verification method, device, storage medium, and product to reduce the time consumption and computational resource consumption of model verification.

[0005] On the one hand, embodiments of this application provide a model verification method, the method comprising: Obtain the evaluation sample set and reference accuracy of the target model, wherein the reference accuracy refers to the model-level accuracy obtained by the target model running on the reference hardware platform by processing the evaluation sample set; Multiple sub-sample sets are extracted from the evaluation sample set, and each sub-sample set corresponds to the first-level verification. At least one level of verification is performed on the target model running on the artificial intelligence chip. The execution process of each level of verification includes the following operations: using the subsample set corresponding to the current level of verification to verify the target model and obtain the preliminary verification result of the current level of verification, wherein the preliminary verification result of the current level of verification includes the verification accuracy; and obtaining the accuracy difference of the current level of verification based on the verification accuracy and the reference accuracy. When the accuracy difference of the current verification exceeds the preset accuracy threshold, the model verification ends, and the verification failure is taken as the target verification result of the target model.

[0006] On one hand, embodiments of this application provide a model verification apparatus, which includes: The acquisition module is used to acquire the evaluation sample set and reference accuracy of the target model, wherein the reference accuracy refers to the model-level accuracy obtained by the target model running on the reference hardware platform by processing the evaluation sample set; The extraction module is used to extract multiple sub-sample sets from the evaluation sample set, each sub-sample set corresponding to a first-level verification; The verification module is used to perform at least one level of verification on the target model running on the artificial intelligence chip. The execution process of each level of verification includes the following operations: using the subsample set corresponding to the current level of verification to verify the target model and obtain the preliminary verification result of the current level of verification, wherein the preliminary verification result of the current level of verification includes verification accuracy; and obtaining the accuracy difference of the current level of verification based on the verification accuracy and the reference accuracy. When the accuracy difference of the current verification exceeds the preset accuracy threshold, the model verification ends, and the verification failure is taken as the target verification result of the target model.

[0007] Optionally, the verification module is further configured to: When the accuracy difference of the current level verification is not greater than the accuracy threshold, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold. When the accuracy margin of the current verification is greater than the preset margin threshold, the model verification ends and the verification pass is taken as the target verification result of the target model.

[0008] Optionally, the verification module is specifically used for: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the first level verification and the last level verification, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold.

[0009] Optionally, the verification module is further configured to: When the accuracy margin of the current verification level is not greater than the preset margin threshold, the next level of verification is executed.

[0010] Optionally, the verification module is further configured to: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the last level verification, the next level verification is executed; When the accuracy difference of the current verification is not greater than the accuracy threshold, and the current verification is the last level of verification, the model verification ends, and the verification pass is taken as the target verification result of the target model.

[0011] Optionally, the verification module is specifically used for: The multiple sub-sample sets are sorted in ascending order of sample size to obtain the sorting result; Based on the sorting results, the execution order of the multi-level verification corresponding to the multiple sub-sample sets is obtained; Perform at least one level of verification on the target model according to the execution order.

[0012] Optionally, the acquisition module is specifically used for: Search for the published data of the target model using a web search engine based on the model identifier of the target model; Extract the evaluation sample set and reference accuracy of the target model from the published data.

[0013] Optionally, the acquisition module is further configured to: When the published data does not include the evaluation sample set and the reference accuracy, the target model family to which the target model belongs is confirmed. Obtain the evaluation benchmark information of the target model family from the preset benchmark mapping table; The evaluation sample set and reference accuracy of the target model are obtained from the evaluation benchmark information.

[0014] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, a central processing unit, and a computer program stored in the memory and running on the central processing unit, wherein the central processing unit executes the computer program to implement the steps of the above-described model verification method.

[0015] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described model verification method.

[0016] On one hand, embodiments of this application provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described model verification method.

[0017] In this embodiment, when migrating the target model from a reference hardware platform to an AI chip, multiple sub-sample sets extracted from the evaluation sample set are used, and a progressive verification strategy is employed to perform verification step by step. During each verification stage, the accuracy difference between the verification accuracy and the reference accuracy is calculated. If the accuracy difference exceeds a preset accuracy threshold, the model verification is terminated directly, and the verification failure is taken as the target verification result for the target model. This allows for rapid identification of accuracy issues in the target model and timely mitigation of losses, avoiding the continued consumption of expensive computing resources when problems are already known, thereby effectively reducing resource waste and decreasing the time required for model accuracy verification.

[0018] This application performs end-to-end inference on an artificial intelligence chip and adopts a progressive verification strategy to perform verification step by step during the inference process to obtain the verification accuracy, which is the model-level task accuracy. Then, based on the accuracy difference between the verification accuracy and the reference accuracy, the model-level verification result is quickly obtained. Compared with verifying the model-level accuracy based on operator-level difference indicators, the accuracy is higher. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating a model verification method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the architecture of an AI agent provided in an embodiment of this application; Figure 4 A flowchart illustrating another model verification method provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of a model verification device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] Under relevant technologies, after migrating AI models from general-purpose chips to self-developed accelerator chips, it is necessary to verify whether the model's accuracy on the new hardware meets the requirements. Currently, the full-scale execution method is commonly used to verify AI models. However, this verification method lacks cost awareness; that is, regardless of whether the model has serious accuracy problems, a complete large-scale benchmark test is performed. This wastes a lot of computing resources when problems are obvious, and it takes several hours to confirm failure after completing the full benchmark test, resulting in a long verification time.

[0023] In addition, some schemes calculate numerical deviation indices for each operator and then use operator-level numerical deviation indices to verify model accuracy. However, model accuracy depends on the synergistic effect of all operators. Low numerical deviations of individual operators cannot guarantee that the end-to-end model accuracy meets the requirements. Conversely, large deviations of operators may be absorbed by the normalization operations of subsequent layers. Therefore, the accuracy of operator-level accuracy verification methods is relatively low.

[0024] In view of this, embodiments of this application provide a progressive cost-aware model verification method. The following first describes a structural diagram of an artificial intelligence chip to which this application embodiment applies, such as... Figure 1 As shown, the artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be streaming processing clusters (SPCs). The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.

[0025] In this embodiment of the application, when the target model is migrated from the reference hardware platform to the artificial intelligence chip 100, the target model is validated by the AI ​​agent in the following manner: The evaluation sample set and reference accuracy of the target model are obtained. The target model can be any artificial intelligence model; the reference accuracy refers to the model-level accuracy obtained by the target model running on the reference hardware platform after processing the evaluation sample set. The reference hardware platform and the artificial intelligence chip 100 are different hardware.

[0026] Multiple subsets of samples are extracted from the evaluation sample set, each subset corresponding to a first-level verification. Then, at least one level of verification is performed on the target model running on the AI ​​chip 100. The execution process of each level of verification includes the following operations: using the subset of samples corresponding to the current level of verification, the target model is verified to obtain the preliminary verification result of the current level of verification, which includes the verification accuracy; then, based on the verification accuracy and the reference accuracy, the accuracy difference of the current level of verification is obtained.

[0027] When the accuracy difference of this level of verification exceeds the preset accuracy threshold, the model verification ends, and the verification failure is taken as the target verification result of the target model.

[0028] In this embodiment, when migrating the target model from a reference hardware platform to an AI chip, multiple sub-sample sets extracted from the evaluation sample set are used, and a progressive verification strategy is employed to perform verification step by step. During each verification stage, the accuracy difference between the verification accuracy and the reference accuracy is calculated. If the accuracy difference exceeds a preset accuracy threshold, the model verification is terminated directly, and the verification failure is taken as the target verification result for the target model. This allows for rapid identification of accuracy issues in the target model and timely mitigation of losses, avoiding the continued consumption of expensive computing resources when problems are already known, thereby effectively reducing resource waste and decreasing the time required for model accuracy verification.

[0029] This application performs end-to-end inference on an artificial intelligence chip and adopts a progressive verification strategy to perform verification step by step during the inference process to obtain the verification accuracy, which is the model-level task accuracy. Then, based on the accuracy difference between the verification accuracy and the reference accuracy, the model-level verification result is quickly obtained. Compared with verifying the model-level accuracy based on operator-level difference indicators, the accuracy is higher.

[0030] In the embodiments of this application, the AI ​​agent can be run by the central processing unit, or the AI ​​agent can be run by both the central processing unit and the artificial intelligence chip. This application does not make any specific limitations on this.

[0031] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0032] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-Purpose Graphics Processing Units (GPGPU), Domain Specific Architecture (DSA), etc.

[0033] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the process of a model verification method. (See attached image.) Figure 2 This method is executed by a computer device, which includes an artificial intelligence chip and a central processing unit. The computer device is equipped with an AI agent, which performs the following steps: Step 201: Obtain the evaluation sample set and reference accuracy of the target model.

[0034] In this application, the target model can be applied to various scenarios, such as image processing, speech processing, and text processing. The physical meaning of the input data processed by the target model can differ depending on the application scenario.

[0035] For example, in a text processing scenario, the input data can be text data used in tasks such as text generation and text recognition.

[0036] For example, in a speech processing scenario, the input data can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis.

[0037] For example, in image processing scenarios, the input data can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0038] In this application, reference accuracy refers to the model-level accuracy obtained by processing the evaluation sample set on the target model running on the reference hardware platform. Since different model families have different evaluation focuses, the evaluation sample set and reference accuracy used for model validation of artificial intelligence models from different model families are also different.

[0039] In the embodiments of this application, the evaluation sample set and reference accuracy of the target model are self-discovered at least in the following ways: The target model's published data is searched using a web search engine based on its model identifier; then, the evaluation sample set and reference accuracy of the target model are extracted from the published data.

[0040] Specifically, the published data includes authoritative publications such as technical reports, model cards, and official evaluation forms. Search results are obtained by using the target model's identifier (e.g., model name) as keywords in a web search. These results are then parsed to obtain the target model's published data. Next, a model evaluation form is extracted from the published data, and it is determined whether the form contains an evaluation benchmark. The benchmark includes the evaluation sample set used at the time of the target model's publication and its corresponding reference accuracy. If so, the name of the evaluation sample set is normalized, and the target model's evaluation sample set and reference accuracy are downloaded when it is determined that the verification evaluation sample set is downloadable. In addition, the evaluation benchmark also includes the identification information of the reference hardware platform extracted from the model evaluation form. The reference hardware platform is the hardware environment used to obtain the reference accuracy.

[0041] In this embodiment, the report data of the target model is queried through a web search engine to automatically extract the evaluation benchmark used when the target model was released. This allows different evaluation benchmarks to be used for accuracy verification for different models, and the model and the evaluation benchmark used for verification are more matched. Compared with the fixed benchmark method (i.e. all models use the same set of fixed evaluation benchmarks), this greatly improves the accuracy of model accuracy verification and no longer relies on human experience to select the verification benchmark.

[0042] In some embodiments, when the published data does not contain an evaluation sample set and reference accuracy, the target model family to which the target model belongs is determined; then the evaluation benchmark information of the target model family is obtained from a preset benchmark mapping table; and then the evaluation sample set and reference accuracy of the target model are obtained from the evaluation benchmark information.

[0043] Specifically, a default benchmark mapping table is maintained according to the model family. Each model family corresponds to an evaluation benchmark set. Each evaluation benchmark set includes at least one evaluation benchmark. Each evaluation benchmark includes: an evaluation sample set and the corresponding reference accuracy.

[0044] For example, see Table 1 below, which is a reference mapping table provided in an embodiment of this application.

[0045] Table 1

[0046] When a search fails, the process falls back to the benchmark mapping table maintained by model family and retrieves the evaluation benchmark set of the target model's family from the benchmark mapping table. When the evaluation benchmark set of the target model family includes multiple evaluation benchmarks, the accuracy of the target model can be verified from at least one evaluation benchmark in the evaluation benchmark set.

[0047] For example, if an evaluation benchmark (i.e., evaluation sample set and corresponding reference accuracy) for the target model cannot be obtained through online search, and the target model belongs to a dense large language model family, then GSM8K in Table 1 can be selected as the evaluation benchmark to validate the target model.

[0048] In this embodiment, the default benchmark mapping table of the model family is used as an intelligent fallback when the search fails, which ensures that an evaluation benchmark is available, thereby improving the stability of model validation.

[0049] Step 202: Extract multiple sub-sample sets from the evaluation sample set.

[0050] Specifically, the physical meaning of samples in a subsample set can differ depending on the application scenario. For example, in a text processing scenario, samples can be text data used in tasks such as text generation and text recognition. In a speech processing scenario, samples can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis. In an image processing scenario, samples can be image data used in tasks such as image preprocessing, image segmentation, and object detection.

[0051] Multiple subsets of samples have different sample sizes, and correspondingly, the computational cost of validating a model using different subsets also varies. The computational cost includes computational resources, storage resources, and time consumption. Generally speaking, the larger the sample size of a subset, the greater the computational cost of validating the model using that subset.

[0052] Based on the computational cost of each subset, the subsets are divided into multiple cost levels, with higher computational costs corresponding to higher cost levels. Each subset corresponds to a first-level validation, and each subset can be used independently for model validation. It should be noted that there can be overlapping samples among the subsets, and of course, the subsets can also be completely different.

[0053] For example, see Table 2 below for information on sub-sample sets of multiple cost levels provided in the embodiments of this application (sample size, time consumption, and verification purpose).

[0054] Table 2

[0055] Step 203: Perform at least one level of verification on the target model, wherein the execution process of each level of verification includes the following steps 2031-2032: Specifically, in scenarios where the target model is migrated from a reference hardware platform to an AI chip, at least one level of verification is performed on the target model running on the AI ​​chip.

[0056] In practical applications, some models with obvious accuracy problems (such as missing operators causing all outputs to be zero) can be quickly identified using a small number of samples without needing to perform a full evaluation sample set. Based on this, this application adopts a progressive verification strategy, performing verification on the target model level by level according to computational cost (i.e., cost level) from low to high. The execution order of multi-level verification corresponding to multiple sub-sample sets is obtained as follows: The multiple subsets are sorted in ascending order of sample size to obtain the ranking result. Since the sample size of a subset is positively correlated with its computational cost, this ranking result also represents the result of sorting multiple subsets in ascending order of computational cost.

[0057] Then, based on the sorting results, the execution order of multi-level validation corresponding to multiple sample sets is obtained, that is, the execution order of multi-level validation is in the order of computational cost from low to high; then, at least one level of validation is performed on the target model according to the execution order.

[0058] In this embodiment, a progressive verification strategy is adopted, and the target model is verified step by step in order of computational cost (i.e. cost level) from low to high. This can quickly identify problems and stop losses in time, thereby effectively reducing resource waste and reducing the time spent on model accuracy verification.

[0059] Step 2031: Use the subsample set corresponding to this level of validation to validate the target model and obtain the preliminary validation results of this level of validation.

[0060] Step 2032: Based on the verification accuracy and the reference accuracy, obtain the accuracy difference of this level of verification.

[0061] Specifically, the target model running in the AI ​​chip processes each sample in the sub-sample set for this level of verification to obtain the corresponding processing results. Then, based on the processing results of each sample, a preliminary verification result for this level of verification is obtained, which includes the verification accuracy. The difference between the verification accuracy and the reference accuracy is then calculated to obtain the accuracy difference for this level of verification.

[0062] Step 204: When the accuracy difference of this level of verification is greater than the preset accuracy threshold, the model verification ends and the verification failure is taken as the target verification result of the target model.

[0063] Specifically, when the accuracy difference of this level of verification is greater than the accuracy threshold, and this level of verification is not the last level of verification, it means that the problems of the target model have been exposed, and there is no need to perform subsequent levels of verification. Therefore, the model verification is terminated directly, and the verification failure is taken as the target verification result of the target model.

[0064] If the accuracy difference of this level of verification is greater than the accuracy threshold, and this level of verification is the last level of verification, then the model verification will end directly, and the verification failure will be taken as the target verification result of the target model.

[0065] If the accuracy difference of this level of verification is not greater than the accuracy threshold, and this level of verification is not the last level of verification, no model problem has been found yet, so the next level of verification needs to be performed.

[0066] If the accuracy difference of this level of verification is not greater than the preset accuracy threshold, and this level of verification is the last level of verification, then the model verification will end directly, and the verification pass will be taken as the target verification result of the target model.

[0067] In this embodiment, when migrating the target model from a reference hardware platform to an AI chip, multiple sub-sample sets extracted from the evaluation sample set are used, and a progressive verification strategy is employed to perform verification step by step. During each verification stage, the accuracy difference between the verification accuracy and the reference accuracy is calculated. If the accuracy difference exceeds a preset accuracy threshold, the model verification is terminated directly, and the verification failure is taken as the target verification result for the target model. This allows for rapid identification of accuracy issues in the target model and timely mitigation of losses, avoiding the continued consumption of expensive computing resources when problems are already known, thereby effectively reducing resource waste and decreasing the time required for model accuracy verification.

[0068] This application performs end-to-end inference on an artificial intelligence chip and adopts a progressive verification strategy to perform verification step by step during the inference process to obtain the verification accuracy, which is the model-level task accuracy. Then, based on the accuracy difference between the verification accuracy and the reference accuracy, the model-level verification result is quickly obtained. Compared with verifying the model-level accuracy based on operator-level difference indicators, the accuracy is higher.

[0069] In some embodiments, when the accuracy difference of the current verification level is not greater than the accuracy threshold, the accuracy margin of the current verification level is obtained based on the verification accuracy of the current verification level and the accuracy threshold; when the accuracy margin of the current verification level is greater than the preset margin threshold, the model verification ends and the verification pass is taken as the target verification result of the target model.

[0070] Specifically, the difference between the verification accuracy of this level and the accuracy threshold is used as the accuracy margin of this level of verification. When the accuracy margin of this level of verification is greater than the preset margin threshold, it indicates that this level of verification has sufficient statistical confidence to show that the model accuracy is safe. Therefore, the subsequent levels of verification are skipped (i.e., the model verification ends), and the verification pass is taken as the target verification result of the target model.

[0071] When the accuracy margin of this level of verification is not greater than the preset margin threshold, no model problem has been found yet, so the next level of verification is executed.

[0072] In this embodiment, when the accuracy margin of the current verification level is greater than the margin threshold, it indicates that the accuracy of the current verification level is sufficient. Therefore, the subsequent higher-level verification is skipped directly, thereby effectively reducing resource consumption and improving the efficiency of model accuracy verification.

[0073] In some embodiments, when the accuracy difference of the current level of verification is not greater than the accuracy threshold, and the current level of verification is a level of verification other than the first level of verification and the last level of verification, the accuracy margin of the current level of verification is obtained based on the verification accuracy of the current level of verification and the accuracy threshold.

[0074] Specifically, since the sample size for Level 1 validation is typically small, the statistical standard error is large, making it unsuitable to skip subsequent higher-level validations based on precision margins. Similarly, since there are no further levels of validation after the final level, it's unnecessary to skip higher-level validations based on precision margins. For other levels of validation besides Level 1 and the final level, the sample size is usually sufficient to support statistical judgment, and higher-level validations follow. Therefore, for these other levels, a corresponding precision margin is calculated, and model validation ends when the precision margin exceeds a preset margin threshold. A successful validation is then taken as the target validation result for the target model.

[0075] It should be noted that for verification levels other than the first and last verification levels, the corresponding precision margin can be calculated for each other verification level, and it can be determined whether to skip subsequent higher-level verifications. Alternatively, the corresponding precision margin can be calculated for some other verification levels, and it can be determined whether to skip subsequent higher-level verifications. This application does not make specific limitations on this.

[0076] In this embodiment, for verification levels other than the first and last verification levels, the corresponding accuracy margin is calculated and it is determined whether to skip subsequent higher-level verifications. This effectively reduces resource consumption and improves the efficiency of model accuracy verification while ensuring the accuracy of model accuracy verification.

[0077] To better explain the embodiments of this application, the following is combined with... Figure 3 This application describes the process of a progressive cost-aware model verification method provided in an embodiment. The method is executed by an AI agent deployed on a computer device. The AI ​​agent includes: a baseline self-discovery module 301, a cost grading module 302, a progressive execution engine 303, and a result evaluation module 304.

[0078] The benchmark self-discovery module 301 is used to obtain the evaluation sample set and reference accuracy of the target model through network search.

[0079] The cost grading module 302 is used to determine three cost levels and the sample size corresponding to each cost level; then, according to the sample size corresponding to each cost level, the evaluation sample set is uniformly stratified and sampled to obtain the sub-sample set corresponding to each cost level.

[0080] The progressive execution engine 303 is used to perform verification step by step from low to high cost level. At each level, stop-loss conditions and skip conditions are set. The stop-loss conditions include: the accuracy difference is greater than a preset accuracy threshold, and the skip conditions include: the accuracy margin is greater than a preset margin threshold.

[0081] Specifically, three cost levels are defined in ascending order: Level 1, Level 2, and Level 3, with each cost level corresponding to a subset of samples. The verification process, performed level by level from low to high, includes the following steps: Step 3031: Perform Level 1 verification to obtain the accuracy difference.

[0082] Step 3032: Determine whether the precision difference is greater than the precision threshold. If yes, proceed to step 3033; otherwise, proceed to step 30310.

[0083] Step 3033: Perform Level 2 verification to obtain the accuracy difference.

[0084] Step 3034: Determine whether the precision difference is greater than the precision threshold. If yes, proceed to step 3035; otherwise, proceed to step 30310.

[0085] Step 3035: Calculate the accuracy margin for level 2.

[0086] Step 3036: Determine whether the precision margin is greater than the margin threshold. If yes, proceed to step 3039; otherwise, proceed to step 3037.

[0087] Step 3037: Perform Level 3 verification to obtain the accuracy difference.

[0088] Step 3038: Determine whether the precision difference is greater than the precision threshold. If yes, proceed to step 3039; otherwise, proceed to step 30310.

[0089] Step 3039: Output the result indicating successful verification.

[0090] Step 30310: Output the result of verification failure.

[0091] The results evaluation module 304 is used to generate a verification report based on the verification results output by the progressive execution engine 303, in order to record the complete process and results of progressive verification.

[0092] The following section details the time-series interaction process for model validation, such as... Figure 4 As shown, it includes the following steps: Step 401: The AI ​​agent obtains the evaluation sample set and reference accuracy of the target model through network search.

[0093] Step 402: The AI ​​agent determines three cost levels and the sample size for each cost level.

[0094] Specifically, the three cost levels are ordered from low to high as follows: Level 1, Level 2 and Level 3. The sample size for Level 1 is 50, the sample size for Level 2 is 500, and the sample size for Level 3 is 5000 (sample size of the evaluation sample set).

[0095] Step 403: The AI ​​agent performs uniform stratified sampling on the evaluation sample set to obtain a sub-sample set for each cost level.

[0096] Step 404: The AI ​​agent initiates Level 1 verification to the evaluation actuator.

[0097] Step 405: The evaluation actuator sends 50 samples to the artificial intelligence chip for inference.

[0098] Step 406: Evaluate the inference results returned by the AI ​​chip received by the actuator.

[0099] Step 407: Evaluate the executor's calculation of verification accuracy based on the inference results.

[0100] Step 408: Evaluate the accuracy returned by the actuator to the AI ​​agent.

[0101] The reference precision is set to 0.85, while the verification precision for Level 1 is 0.88. That is, the verification precision for Level 1 is greater than the reference precision, so step 409 is executed.

[0102] Step 409: The verification accuracy is greater than the reference accuracy. The AI ​​agent initiates Level 2 verification to the evaluation actuator.

[0103] Step 410: The evaluation actuator sends 500 samples to the artificial intelligence chip for inference.

[0104] Step 411: Evaluate the inference results returned by the AI ​​chip received by the actuator.

[0105] Step 412: Evaluate the actuator's calculation of verification accuracy based on the inference results.

[0106] Step 413: Evaluate the accuracy returned by the actuator to the AI ​​agent.

[0107] The verification precision for Level 2 is set to 0.91, which means that the verification precision for Level 2 is greater than the reference precision. Therefore, we continue with step 414.

[0108] Step 414: The AI ​​agent calculates the accuracy margin between the verification accuracy and the accuracy threshold.

[0109] The margin threshold is set to 0.05, and the margin between the verification accuracy and the reference accuracy of Level 2 is 0.06, i.e. 0.91-0.85=0.06. At this time, the accuracy margin of Level 2 is greater than the margin threshold, so step 415 is executed.

[0110] Step 415: Since the accuracy margin is greater than the margin threshold, the verification result is output.

[0111] In this embodiment, when migrating the target model from a reference hardware platform to an AI chip, multiple sub-sample sets extracted from the evaluation sample set are used, and a progressive verification strategy is employed to perform verification step by step. During each verification stage, the accuracy difference between the verification accuracy and the reference accuracy is calculated. If the accuracy difference exceeds a preset accuracy threshold, the model verification is terminated directly, and the verification failure is taken as the target verification result for the target model. This allows for rapid identification of accuracy issues in the target model and timely mitigation of losses, avoiding the continued consumption of expensive computing resources when problems are already known, thereby effectively reducing resource waste and decreasing the time required for model accuracy verification.

[0112] This application performs end-to-end inference on an artificial intelligence chip and adopts a progressive verification strategy to perform verification step by step during the inference process to obtain the verification accuracy, which is the model-level task accuracy. Then, based on the accuracy difference between the verification accuracy and the reference accuracy, the model-level verification result is quickly obtained. Compared with verifying the model-level accuracy based on the operator-level difference index, it is more accurate and solves the fundamental problem that operator-level accuracy is not equal to model-level accuracy.

[0113] Based on the same technical concept, this application provides a schematic diagram of the structure of a model verification device, such as... Figure 5 As shown, the model verification device 500 includes: The acquisition module 501 is used to acquire the evaluation sample set and reference accuracy of the target model, wherein the reference accuracy refers to the model-level accuracy obtained by the target model running on the reference hardware platform by processing the evaluation sample set; Extraction module 502 is used to extract multiple sub-sample sets from the evaluation sample set, each sub-sample set corresponding to a first-level verification; The verification module 503 is used to perform at least one level of verification on the target model running on the artificial intelligence chip. The execution process of each level of verification includes the following operations: using the subsample set corresponding to the current level of verification to verify the target model and obtain the preliminary verification result of the current level of verification, wherein the preliminary verification result of the current level of verification includes the verification accuracy; and obtaining the accuracy difference of the current level of verification based on the verification accuracy and the reference accuracy. When the accuracy difference of the current verification exceeds the preset accuracy threshold, the model verification ends, and the verification failure is taken as the target verification result of the target model.

[0114] Optionally, the verification module 503 is further configured to: When the accuracy difference of the current level verification is not greater than the accuracy threshold, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold. When the accuracy margin of the current verification is greater than the preset margin threshold, the model verification ends and the verification pass is taken as the target verification result of the target model.

[0115] Optionally, the verification module 503 is specifically used for: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the first level verification and the last level verification, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold.

[0116] Optionally, the verification module 503 is further configured to: When the accuracy margin of the current verification level is not greater than the preset margin threshold, the next level of verification is executed.

[0117] Optionally, the verification module 503 is further configured to: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the last level verification, the next level verification is executed; When the accuracy difference of the current verification is not greater than the accuracy threshold, and the current verification is the last level of verification, the model verification ends, and the verification pass is taken as the target verification result of the target model.

[0118] Optionally, the verification module 503 is specifically used for: The multiple sub-sample sets are sorted in ascending order of sample size to obtain the sorting result; Based on the sorting results, the execution order of the multi-level verification corresponding to the multiple sub-sample sets is obtained; Perform at least one level of verification on the target model according to the execution order.

[0119] Optionally, the acquisition module 501 is specifically used for: Search for the published data of the target model using a web search engine based on the model identifier of the target model; Extract the evaluation sample set and reference accuracy of the target model from the published data.

[0120] Optionally, the acquisition module 501 is further configured to: When the published data does not include the evaluation sample set and the reference accuracy, the target model family to which the target model belongs is confirmed. Obtain the evaluation benchmark information of the target model family from the preset benchmark mapping table; The evaluation sample set and reference accuracy of the target model are obtained from the evaluation benchmark information.

[0121] In this embodiment, when migrating the target model from a reference hardware platform to an AI chip, multiple sub-sample sets extracted from the evaluation sample set are used, and a progressive verification strategy is employed to perform verification step by step. During each verification stage, the accuracy difference between the verification accuracy and the reference accuracy is calculated. If the accuracy difference exceeds a preset accuracy threshold, the model verification is terminated directly, and the verification failure is taken as the target verification result for the target model. This allows for rapid identification of accuracy issues in the target model and timely mitigation of losses, avoiding the continued consumption of expensive computing resources when problems are already known, thereby effectively reducing resource waste and decreasing the time required for model accuracy verification.

[0122] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0123] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 6 As shown, it includes at least one artificial intelligence chip 100, and a memory 601 and a central processing unit 602 connected to the at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 601 and central processing unit 602 is not limited. Figure 6 Taking the AI ​​chip 100, memory 601, and central processing unit 602 as an example, they are connected via a bus. The bus can be divided into address bus, data bus, control bus, etc.

[0124] In this embodiment of the application, the memory 601 stores instructions that can be executed by the central processing unit 602 to perform the steps of the above-described model verification method.

[0125] The central processing unit 602 is the control center of the computer device. It can connect to various parts of the computer device through various interfaces and lines, and realize model verification by running or executing instructions stored in memory 601 and calling data stored in memory 601.

[0126] In some embodiments, the central processing unit 602 and the memory 601 may be implemented on the same chip, while in other embodiments they may be implemented on separate chips.

[0127] Memory 601, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 601 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory 601 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 601 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0128] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the above-described model verification method.

[0129] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the above-described model verification method.

[0130] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0135] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A model validation method, characterized in that, include: Obtain the evaluation sample set and reference accuracy of the target model, wherein the reference accuracy refers to the model-level accuracy obtained by the target model running on the reference hardware platform by processing the evaluation sample set; Multiple sub-sample sets are extracted from the evaluation sample set, and each sub-sample set corresponds to the first-level verification. At least one level of verification is performed on the target model running on the artificial intelligence chip. The execution process of each level of verification includes the following operations: using the subsample set corresponding to the current level of verification to verify the target model and obtain the preliminary verification result of the current level of verification, wherein the preliminary verification result of the current level of verification includes the verification accuracy; and obtaining the accuracy difference of the current level of verification based on the verification accuracy and the reference accuracy. When the accuracy difference of the current verification exceeds the preset accuracy threshold, the model verification ends, and the verification failure is taken as the target verification result of the target model.

2. The method as described in claim 1, characterized in that, Also includes: When the accuracy difference of the current level verification is not greater than the accuracy threshold, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold. When the accuracy margin of the current verification is greater than the preset margin threshold, the model verification ends and the verification pass is taken as the target verification result of the target model.

3. The method as described in claim 2, characterized in that, When the accuracy difference of the current level verification is not greater than the accuracy threshold, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold, including: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the first level verification and the last level verification, the accuracy margin of the current level verification is obtained based on the verification accuracy of the current level verification and the accuracy threshold.

4. The method as described in claim 2, characterized in that, Also includes: When the accuracy margin of the current verification level is not greater than the preset margin threshold, the next level of verification is executed.

5. The method as described in claim 1, characterized in that, Also includes: When the accuracy difference of the current level verification is not greater than the accuracy threshold, and the current level verification is a level verification other than the last level verification, the next level verification is executed; When the accuracy difference of the current verification is not greater than the accuracy threshold, and the current verification is the last level of verification, the model verification ends, and the verification pass is taken as the target verification result of the target model.

6. The method according to any one of claims 1 to 5, characterized in that, Performing at least one level of verification on the target model includes: The multiple sub-sample sets are sorted in ascending order of sample size to obtain the sorting result; Based on the sorting results, the execution order of the multi-level verification corresponding to the multiple sub-sample sets is obtained; Perform at least one level of verification on the target model according to the execution order.

7. The method as described in any one of claims 1 to 5, characterized in that, The acquisition of the evaluation sample set and reference accuracy of the target model includes: Search for the published data of the target model using a web search engine based on the model identifier of the target model; Extract the evaluation sample set and reference accuracy of the target model from the published data.

8. The method as described in claim 7, characterized in that, Also includes: When the published data does not include the evaluation sample set and the reference accuracy, the target model family to which the target model belongs is confirmed. Obtain the evaluation benchmark information of the target model family from the preset benchmark mapping table; The evaluation sample set and reference accuracy of the target model are obtained from the evaluation benchmark information.

9. A computer device, comprising a memory, an artificial intelligence chip, a central processing unit (CPU), and a computer program stored in the memory and running on the CPU, characterized in that, When the central processing unit executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that is executed by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer device, cause the computer device to perform the steps of the method as described in any one of claims 1-8.

Images