Model precision problem positioning method, electronic equipment, storage medium and program product

Abstract

The invention discloses a model precision problem positioning method, an electronic device, a storage medium and a program product, and relates to the technical field of artificial intelligence. According to multi-level tensor data in a task execution process of a target model, the precision problem in the task execution process of the target model is detected level by level; omission of a certain calculation scene problem in the task execution process of the target model is avoided, the reliability and comprehensiveness of problem positioning are improved, a current-level detection result is used for assisting detection of a next-level precision problem in the problem positioning process, and when it is found that a certain level is abnormal, the accuracy of the next-level precision problem is improved. According to the method, the abnormal part of the level is directly and deeply analyzed, comprehensive troubleshooting of the whole target model is not needed, and then the problem positioning efficiency can be improved, so that the technical problems of low precision problem positioning accuracy and positioning efficiency in the task execution process of the model in the related technology can be solved, and the user experience is improved. The technical effects of improving the accuracy and efficiency of model precision problem positioning and the like are achieved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for locating model accuracy problems, electronic equipment, storage media and program products. Background Art

[0002] In the process of model execution, the accuracy of the execution process often fails to meet expectations. In the related technology, the accuracy problem of the model execution process is located based on the operator unit test. The operator unit test module is used to test the test results of the operator in various test scenarios, and the CPU (Central Processing Unit) results are used for verification. If the difference is large, it is considered that the operator has an accuracy problem in this scenario.

[0003] However, the test scenarios in related technologies are mainly constructed based on the experience of engineers, and it is easy to miss extreme computing scenarios when the model is executing tasks, which makes it difficult to locate accuracy problems during task execution. In order to locate the computing problems of some operators, all operators usually need to be checked, resulting in low efficiency in locating accuracy problems during task execution. Summary of the invention

[0004] The present invention provides a model precision problem positioning method, electronic equipment, storage medium and program product, so as to at least solve the problem of low precision problem positioning accuracy and positioning efficiency in the process of model execution task in the related art.

[0005] The present invention provides a method for locating model accuracy problems, comprising the following steps: acquiring multi-level tensor data of a target model in the process of executing a target task; performing step-by-step detection of accuracy problems in the process of the target model executing the target task based on the multi-level tensor data, wherein the detection result of the current level is used to assist the detection of accuracy problems in the next level; and locating the position of the accuracy problem in the process of the model executing the target task based on the detection result of the last level.

[0006] The present invention also provides a device for locating model accuracy problems, including: an acquisition module, used to acquire multi-level tensor data in the process of a target model executing a target task; a detection module, used to perform step-by-step detection of accuracy problems in the process of a target model executing a target task based on the multi-level tensor data, wherein the detection results of the current level are used to assist in the detection of accuracy problems in the next level; and a positioning module, used to locate the position of accuracy problems in the process of a model executing a target task based on the detection results of the last level.

[0007] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned model accuracy problem locating methods when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned model accuracy problem locating methods are implemented.

[0009] The present invention also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned model accuracy problem locating methods when the computer program is executed by a processor.

[0010] Through the present invention, according to the multi-level tensor data of the target model in the process of executing the target task, the precision problems in the process of executing the target task of the target model are detected step by step, which can avoid the omission of a certain computing scenario problem in the process of executing the target task of the target model, improve the reliability and comprehensiveness of problem location, and then locate the position of the precision problem in the model executing the target task according to the detection result of the last level, and use the detection result of the current level to assist the detection of the precision problem of the next level in the problem location process. When an abnormality is found in the tensor data of a certain level, the abnormal part of this level can be directly analyzed in depth without conducting a comprehensive and indiscriminate problem investigation on the entire target model, thereby improving the efficiency and accuracy of problem location. Therefore, the technical problems of poor accuracy and low efficiency in locating precision problems in the process of the model executing the target task in the related technology can be solved, and the technical effects of improving the accuracy and efficiency of locating precision problems in the process of the model executing the target task can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A flow chart of a method for locating model accuracy problems provided by an embodiment of the present invention; Figure 2 A flowchart of a method for locating accuracy issues in large language model training provided by an embodiment of the present invention; Figure 3 A schematic diagram of a device for locating model accuracy problems provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] Before describing the solution of the present invention, some related technologies involved in the present invention are described to assist in understanding the solution of the present invention.

[0016] LLM (Large Language Model) is an artificial intelligence model designed to understand and generate human language. It generally contains language models with tens of billions or even more parameters. These models are trained on large amounts of text data to obtain general large models. General large models are powerful, but may not perform well in specific areas. In order to enable large models to better adapt to and complete tasks in specific areas, the trained general large models can be fine-tuned using data sets in specific areas to give the general large models more customized functions. However, there is currently a problem of difficulty in adapting large models when executing tasks on certain processors.

[0017] To this end, Related Technology 1 proposes an AI computing power execution framework that can greatly reduce the difficulty of different processors adapting to large model execution tasks. However, due to the lack of maturity of some processors or defects in their docking with the framework, various problems are easily introduced in practical applications. These problems will not only affect the accuracy of large model execution tasks, but may also cause precision deviations in the calculation process. Therefore, it is necessary to troubleshoot these potential problems to ensure the correctness of large model execution tasks.

[0018] Since large models can be broken down into thousands of independent operator computing tasks during task execution, accuracy issues often stem from errors in the calculation of some of the operators. Therefore, the accuracy issues in related technology 1 can be located by detecting the operators. To this end, related technology 2 locates accuracy issues based on operator unit testing, by creating as comprehensive a test scenario as possible for each operator supported by the artificial intelligence computing power execution framework, and verifying the test results of the operator in various scenarios with the CPU results. If the difference is large, it is considered that the operator has accuracy issues in this scenario.

[0019] However, the related technology 2 tests all operators supported by the AI ​​computing power execution framework, which is a wide-area troubleshooting method suitable for rough screening of problematic operators when various processors are just connected to the framework. However, for locating accuracy problems in large models in specific fields when executing certain tasks, there are the following problems: 1. Problem scenarios are easily missed: The completeness of the test scenario depends largely on the experience of engineers. The model has a huge number of parameters and a complex structure. After taking a series of optimization measures such as parallelization and splitting, uncommon extreme computing scenarios may appear, such as strange tensor shapes and memory layouts. These scenarios are easily missed in unit testing, making it difficult to locate accuracy problems of large models.

[0020] 2. Low efficiency in locating operators: The types of operators supported by the AI ​​computing execution framework are generally far more than those used by a large model in a specific field to perform a task. It is inefficient to check all operators in order to locate the computational problems of some operators.

[0021] To this end, the present invention provides a method for locating model accuracy problems to solve the above-mentioned problems.

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0023] An embodiment of the present invention provides a method for locating model accuracy problems.

[0024] like Figure 1 As shown, the model accuracy problem location method includes the following steps: In step S101, multi-level tensor data of the target model in the process of executing the target task is obtained.

[0025] Among them, the target task can be an inference task or a training task, and the embodiments of the present invention focus on describing it in terms of training tasks; the target model can be an LLM; tensor data is the data flow in the process of the target model executing the task, which runs through the entire task execution process. The model execution task involves the calculation of hundreds or thousands of operators at a time. The data transmitted and processed in these calculation processes is tensor data. Starting from the input data of the large model, through the calculation of each module, layer and operator, the data always exists and flows in the form of tensors, which carry various information required for the model to execute the task and are the basis for the model to perform calculations and parameter updates.

[0026] In an embodiment of the present invention, multi-level tensor data is obtained during the process of a target model executing a target task, including: identifying a model structure of the target model; dividing the target model into multiple levels of targets to be extracted according to the model structure; embedding a tensor extraction function before and after the targets to be extracted in the multiple levels of targets to be extracted; and using the tensor extraction function to extract multi-level tensor data of the multiple levels of targets to be extracted.

[0027] Among them, the tensor extraction function can be implemented through the data saving method of the PyTorch framework (i.e. torch.save()).

[0028] It can be understood that the embodiments of the present invention can divide the target model into multiple layers of targets to be extracted according to the model structure, and embed tensor extraction functions before and after the targets to be extracted in the multi-level targets to be extracted, and use the tensor extraction function to extract the multi-level tensor data of the multi-level extraction targets, so as to realize the extraction of tensor data at each level, so as to solve the accuracy problems in the subsequent positioning model executing the target task.

[0029] In an embodiment of the present invention, the target model is divided into multiple levels of targets to be extracted according to the model structure, including: determining the multi-level data granularity of the target model according to the model structure; and dividing the target model into multiple levels of targets to be extracted according to the multi-level data granularity.

[0030] Among them, data granularity refers to the hierarchical level at which data is divided and observed when analyzing and processing the target model. The target model has a clearly hierarchical structure, from the overall model, to each module in the model, to each layer in the module, and finally to each operator in the layer, forming different levels. The data granularity of the current level is greater than that of the next level, that is, the current level refers to a relatively higher level in the target model structure, and the next level is a more subdivided level than the current level. For example, if the current level in the large language model is a module, the next level is a layer.

[0031] It can be understood that the embodiments of the present invention can determine the multi-level data granularity of the target model according to the model structure, and divide the target model into multi-level targets to be extracted according to the multi-level data granularity, so that the subsequent positioning of model accuracy problems through multi-level tensor extraction is more efficient. The investigation can be carried out in order from the whole to the part, from the macro to the micro, and from the coarse granularity to the fine granularity. Starting from the overall data granularity of the model, the modules that may have problems are located, and then the layers within the modules are deeply rooted, and finally the specific operator granularity is accurate. The accuracy problems in the process of the model executing the target task can be quickly located, which greatly reduces the scope of investigation and time cost.

[0032] In an embodiment of the present invention, a tensor extraction function is used to extract multi-level tensor data of multi-level targets to be extracted, including: using a tensor extraction function to extract tensors of multi-level targets to be extracted; obtaining names and serial numbers of multi-level targets to be extracted; and generating multi-level tensor data according to the tensors, names and serial numbers of multi-level targets to be extracted.

[0033] Among them, tensor is a multidimensional array structure used to represent and transmit data in the model calculation process. As a carrier of data, tensor flows between different modules, layers and operators, and participates in various computing operations. Tensor can be in the form of multiple arrays, such as a two-dimensional tensor. A two-dimensional tensor with a shape of (3, 4) can be represented as a matrix with 3 rows and 4 columns.

[0034] It is understandable that the embodiments of the present invention can utilize tensor extraction functions to extract tensors of multi-level targets to be extracted, and generate multi-level tensor data of the targets to be extracted according to the names, sequences and tensors of the multi-level targets to be extracted.

[0035] For example, if the target to be extracted is the first embedding block of a module in a large language model, the sequence number of the embedding block is 1, and the tensor is A, then the corresponding tensor data may be in the form of embedding block 1A.

[0036] The process of tensor extraction in the embodiment of the present invention is described below through a specific embodiment, taking a common large language model as an example.

[0037] The common large language model network structure mainly consists of an embedding block, multiple repeated decoding blocks, and a normalization block. The decoding block is generally repeated dozens of times. For example, the Llama2 model has 32 decoding layers. The decoding block can be composed of a normalization layer, a multi-head self-attention layer, and a fully connected layer. These layers are composed of many various operator calculations, such as matrix multiplication, addition, etc. The network structure of a large language model is as follows: Figure 2 As shown, the large language model structure can be divided into three levels, namely the block level, the layer level and the operator level, which can also be understood as the first level, the second level and the third level.

[0038] Based on Figure 2 The model structure has clear middle-level structures. In the first-level tensor extraction, the tensor extraction function is embedded before and after each module of each large model. For example, the tensor extraction function is embedded before and after the embedding block. The decoding block, normalization block and other modules are similar. The extracted tensor data is named by the module name and module number. In the secondary tensor extraction, tensor extraction functions are embedded before and after each layer of each module. For example, tensor extraction functions are embedded before and after the normalization layer of the decoding block. The multi-head self-attention layer, normalization layer, fully connected layer and other layers are similar. The extracted tensor data is named by the layer name and layer number. In the three-level tensor extraction, tensor extraction functions are embedded before and after each operator in each layer. For example, tensor extraction functions are embedded before and after the matrix multiplication operator of the multi-head self-attention layer. Operators such as the addition operator are similar. The extracted tensor data is named by the operator name and the operator serial number.

[0039] In step S102, the accuracy problems in the process of the target model executing the target task are detected step by step according to the multi-level tensor data, wherein the detection results of the current level are used to assist the detection of the accuracy problems of the next level.

[0040] It can be understood that the embodiments of the present invention can perform step-by-step detection on the accuracy problems in the process of the target model executing the target task according to the multi-level tensor data. Through the step-by-step detection, the detection can be performed in order from the whole to the part, from the coarse granularity to the fine granularity, which can avoid the omission of a certain computing scenario problem in the process of the target model executing the target task, and can more quickly locate the accuracy problem in the process of the model executing the target task, thereby reducing the scope of investigation and time cost. In addition, the embodiments of the present invention can also use the detection results of the current level to assist the detection of accuracy problems at the next level, that is, when an abnormality is found in the data of a certain level, the target to be extracted at that level can be directly analyzed in depth, without comprehensive and indiscriminate problem investigation of the entire target model, thereby improving the efficiency and accuracy of problem location.

[0041] In an embodiment of the present invention, accuracy problems in the process of a target model executing a target task are detected step by step based on multi-level tensor data, including: extracting current-level tensor data from the multi-level tensor data in a problem processor and a calibration processor respectively; comparing the current-level tensor data extracted by the problem processor and the calibration processor respectively; and determining the detection result of the current level based on the data comparison result of the problem processor and the calibration processor.

[0042] Among them, the problem processor refers to a processor that encounters model accuracy problems when the target model performs the target task and whose performance does not meet the expected standards. The calibration processor is a processor used to calibrate, correct or adjust the performance relative to other processors.

[0043] It can be understood that the embodiments of the present invention can perform tensor extraction on the problem processor and the calibration processor, and compare the current-level tensor data on the problem processor and the calibration processor to determine the current-level detection results based on the data comparison results, thereby helping researchers to optimize and improve the problem processor in a targeted manner, thereby improving the performance and adaptability of the processor in the process of executing the target task of the model.

[0044] In an embodiment of the present invention, based on the data comparison results of the problem processor and the calibration processor, the detection result of the current level is determined, including: identifying data errors in the data comparison results; if the data error is less than the problem threshold, the detection result of the current level is passed; if the data error is greater than or equal to the problem threshold, the detection result of the current level is failed.

[0045] Among them, the problem threshold can be set according to the specific situation and there is no specific limitation on this.

[0046] It can be understood that the embodiments of the present invention can identify data errors in data comparison results. If the data error is less than the problem threshold, the detection result of the current level of the problem processor passes, indicating that there is no problem with the accuracy of the target model in the process of executing the target task at the current level of the problem processor. If the data error is greater than or equal to the problem threshold, the detection result of the current level is that the detection fails, indicating that there is a problem with the accuracy of the target model in the process of executing the target task at the current level of the problem processor. By comparing the data error with the problem threshold, the uncertainty of human subjective judgment is avoided, and the detection result can be made more accurate.

[0047] For example, the current level tensor extraction is performed on the problem processor and the calibration processor for the large language model training process, that is, the current level is the first level, that is, the module level. Among them, the tensor extraction can use the tensor comparison method of the PyTorch framework (i.e. torch.testing.assert_close()) to loop and compare the errors of a series of tensor data saved on the problem processor and the calibration processor respectively. If the data error is less than the problem threshold, the first-level detection result of the problem processor passes, otherwise the first-level detection result of the problem processor fails.

[0048] In an embodiment of the present invention, the detection results of the current level are used to assist the detection of precision problems at the next level, including: extracting problem targets that failed the detection in the detection results of the current level; and assisting the detection of precision problems at the next level based on the problem targets that failed the detection.

[0049] It can be understood that the embodiments of the present invention can extract the problem targets that failed the detection in the current level of detection results, and assist in the detection of precision problems at the next level based on the problem targets that failed the detection, that is, the problem targets that failed the current level of detection are used as the focus of the next level of detection, avoiding indiscriminate detection of all targets, and performing subsequent detections more specifically, so as to locate the precision problem position more quickly and improve the efficiency of problem location.

[0050] For example, the current level is level one (module level), that is, the module level, and the problem target that failed the detection result in level one is the decoding block. Then the decoding block is used to assist in the detection of the accuracy problem of the next level, that is, the detection of the accuracy problem of the second level (level).

[0051] In an embodiment of the present invention, assisting in the detection of the next level of precision problems based on the problem target that failed the detection includes: identifying the identifier of the problem target that failed the detection; locating the position of the problem target based on the identifier of the problem target; and performing the next level of precision problem detection on the target at the position of the problem target.

[0052] The identifier is used to indicate the location of the target, and may include, for example, the name and serial number of the target, which is unique.

[0053] It can be understood that the embodiments of the present invention can identify the identifier of the problem target that failed the detection, and locate the position of the problem target based on the identifier of the problem target, and then perform the next level of precision problem detection on the target at the position of the problem target, and can specifically determine the next level of precision problem detection based on the position of the problem target, and improve the efficiency of problem locating by determining the position of the precision problem step by step.

[0054] In the embodiment of the present invention, locating the position of the problem target according to the identifier of the problem target includes: determining the name and serial number of the problem target according to the identifier of the problem target; and determining the position of the problem target according to the name and serial number of the problem target.

[0055] It is understandable that the embodiment of the present invention can determine the name and sequence of the problem target according to the identifier of the problem target, and further determine the position of the problem target according to the name and sequence number of the problem target.

[0056] For example, if the identifier of the problem target is "multi-head attention layer_3", it can be determined that the name of the problem target is multi-head attention layer, and the sequence number is 3, indicating that the data of the third multi-head attention layer in the target model is abnormal.

[0057] In step S103, the location of the accuracy problem in the process of the model performing the target task is located according to the detection result of the last level.

[0058] It can be understood that the embodiments of the present invention can locate the position of the accuracy problem in the process of the model executing the target task according to the detection results of the last level, and locate the accuracy problem of the model in the process of executing the target task to the smallest granularity, that is, the last level, by detecting the multi-level tensor data in the target model step by step, from the whole to the local, from the coarse granularity to the fine granularity. This can avoid the omission of a certain computing scenario problem in the process of the target model executing the target task, and can more quickly locate the accuracy problem in the process of the model executing the target task, reducing the scope of investigation and time cost.

[0059] In an embodiment of the present invention, the location of the accuracy problem in the process of executing the target task by locating the model according to the detection results of the last level includes: extracting the problem target that failed the detection in the detection results of the last level; identifying the identifier of the problem target that failed the detection; and locating the location of the problem target according to the identifier of the problem target.

[0060] It can be understood that the embodiments of the present invention can extract the problem targets that failed the detection in the detection results of the last level, and identify the identifiers of the problem targets that failed the detection, and locate the position of the problem targets according to the identifiers of the problem targets, so as to achieve the minimum level of accuracy in the process of the positioning model executing the target task, wherein the method of locating the position of the problem targets has been explained in the above embodiments and will not be repeated here.

[0061] In an embodiment of the present invention, after locating the location of the accuracy problem in the process of the model executing the target task according to the detection results of the last level, it also includes: obtaining data of the problem target in the last level; and generating an error case based on the data of the problem target.

[0062] Among them, data includes tensor shape, data type, memory layout, etc.

[0063] It can be understood that the embodiments of the present invention can obtain the data of the problem target in the last level, and generate error use cases based on the data of the problem target, so as to be used for subsequent repair of the problem target, so as to improve the accuracy of the subsequent model in executing the target task, thereby improving the adaptability and performance of the processor for the model to execute the target task.

[0064] In an embodiment of the present invention, the data of the problem target includes at least one of a tensor shape, a data type, and a memory arrangement. After an error case is generated according to the data of the problem target, it also includes: providing the error case to a target terminal of a downstream maintenance engineer; at the target terminal, extracting at least one of the tensor shape, data type, and memory arrangement in the error case based on a data reading method of a PyTorch framework; and repairing and testing using at least one of the tensor shape, data type, and memory arrangement.

[0065] The data reading method may be torch.load().

[0066] It is understandable that the embodiments of the present invention can provide error cases to downstream maintenance engineers. The maintenance engineers can extract at least one of the tensor shape, data type and memory layout in the error case based on the data reading method of the PyTorch framework, and use the data reading method to efficiently extract key information such as tensor shape, data type and memory layout, thereby avoiding the tedious process of manually searching and organizing data, thereby significantly improving the efficiency of problem solving. Moreover, since different tensor shapes, data types and memory layouts will affect the calculation logic and results of the operator, by obtaining this information, the maintenance engineer can adjust the algorithm, optimize the code or fix the error in a targeted manner, thereby improving the accuracy of the repair and reducing repeated testing and debugging work caused by incomplete error repair.

[0067] It should be noted that PyTorch is an open source deep learning framework based on Python. PyTorch uses tensors as the core data structure and provides a large number of rich and flexible tensor operation functions, such as creation, transformation, and calculation. When locating precision problems in the process of processing model sub-execution target tasks, various operations and analyses need to be performed on multi-level tensor data. PyTorch can easily meet these needs. For example, it can easily perform dimension transformation and element calculation on the extracted tensors to analyze the characteristics and differences of tensor data. In addition, PyTorch uses dynamic calculation graphs, which means that the calculation graph is dynamically constructed according to the execution path of the code at runtime. When locating precision problems in the process of model execution target tasks, this feature allows maintenance engineers to track and analyze the calculation process of the model more flexibly, and can easily view the calculation results of each step, the changes in tensors, etc., which helps to quickly locate which part of the model calculation or which layer of the network the problem is in. Dynamic calculation graphs make it easier to modify and adjust the model after the precision problem is found. Maintenance engineers can modify the structure or parameters of the model directly in the code according to the analysis results, and then re-run it for verification, without the need to rebuild the entire calculation graph like a static calculation graph, thereby improving the efficiency of problem solving. Therefore, an embodiment of the present invention adopts a data reading method based on the PyTorch framework to read error use cases, so as to better locate the problem and improve the efficiency of problem solving.

[0068] Below Figure 2 Taking the location of model training accuracy problems using the large language model shown as an example, the overall execution process of the model accuracy problem location method of the present invention is described. The large model training accuracy problems in a specific field can be efficiently and quickly located to the operator granularity, greatly improving the efficiency of repairing computational errors of various processors in the artificial intelligence computing power execution framework on large model tasks.

[0069] Among them, the network structure of the large language model (abbreviated as large model) is mainly composed of embedding blocks, multiple repeated decoding blocks, and normalization blocks. The decoding blocks are generally repeated dozens of times. For example, the Llama2 (LargeLanguage Model Augmented with Meta-learned Approaches) model with 7 billion parameters has 32 decoding layers. The decoding block can be composed of a normalization layer, a multi-head self-attention layer, and a fully connected layer, and each layer is composed of many various operator calculations, such as matrix multiplication, addition, etc. Model training involves the calculation of hundreds or thousands of operators. When the training accuracy of the large model is not correct, it means Figure 2 The model output in is wrong.

[0070] In order to quickly locate calculation errors from hundreds or thousands of operators, the present invention adopts a strategy that combines multi-level tensor extraction and automated tensor detection, and conveniently provides operator error use cases so that downstream operator engineers can quickly troubleshoot and fix problems. The specific process includes: Step 1: Multi-level tensor extraction.

[0071] based on Figure 2 The model structure features of the medium and large models with distinct hierarchies, the present invention implements multi-level tensor extraction by inserting tensor extraction functions. The data flow during model training is tensor data, and the extraction function is implemented through the data saving method of the PyTorch framework (i.e., torch.save()). In the first-level tensor extraction, the tensor extraction function is embedded before and after each module of the large model, and the extracted data is saved locally, named by the module name and the module serial number. In the second-level tensor extraction, the tensor extraction function is embedded before and after each layer of the module, and the extracted data is saved locally, named by the layer name and the layer serial number. In the third-level tensor extraction, the tensor extraction function is embedded before and after each operator of the layer, and the extracted data is saved locally, named by the operator name and the operator serial number.

[0072] Step 2: Automated tensor detection.

[0073] Each level of tensor extraction will save a series of tensor data, and in large model tasks, these tensors are often large in shape and have a lot of data, which cannot be compared manually. Based on the known model structure, the tensor names saved in each level of extraction steps in step 1 are generated in advance and configured in the detection script. The first level of tensor extraction is performed on the problem chip and the calibration chip, and then the tensor comparison method of the PyTorch framework (i.e., torch.testing.assert_close()) is used to loop and compare the errors of the series of data saved by the two to achieve automatic tensor detection. The problem module is quickly located according to the module name and serial number in the tensor name that failed the test. Then, the problem module is extracted and compared on the problem processor and calibration processor. The layer name and serial number in the tensor name that failed the test can quickly expose the problem layer. Finally, the problem layer is extracted and compared on the problem processor and calibration processor. The layer name and serial number in the tensor name that failed the test can quickly expose the problem operator.

[0074] The embodiment of the present invention combines multi-level tensor extraction and automated tensor detection. The first-level tensor extraction and detection reduces the accuracy problem from the model granularity to the module granularity, the second-level tensor extraction and detection reduces the problem from the module granularity to the layer granularity, and the third-level tensor extraction and detection reduces the problem from the layer granularity to the operator granularity. Each level of processing can reduce the number of operators to be checked by an order of magnitude. This multi-level troubleshooting strategy greatly reduces the workload of locating accuracy problems and is applicable to any large model.

[0075] Step 3: Operator error use case provided.

[0076] The data saved in the third-level tensor extraction and not passed the automated test is the input and output data of the problematic operator. This data saves various information such as tensor shape, data type, memory layout, etc. required for operator testing, and can be directly provided to downstream operator maintenance engineers. Operator engineers can quickly start operator repair and testing by extracting the required information based on the data reading method of the PyTorch framework (i.e., torch.load()). In addition, the error test case provided by the present invention can also be supplemented to the unit test of the prior art to improve the comprehensiveness of the framework operator unit test.

[0077] In summary, the model accuracy problem location method of the present invention can be used for processor function maintenance in the artificial intelligence computing power execution framework. Among them, the artificial intelligence computing power execution framework aims to facilitate more processors to adapt to large model training or reasoning through a set of general platforms and standards, and has been implemented on some processors. With the increasing number of processor types connected to the artificial intelligence computing power execution framework, it is essential to evaluate the correctness and efficiency of model training and reasoning of these processors in a large number of different scenarios. At the same time, users can also start benchmark evaluations of hundreds of scenarios with one click. In order to promote the common development of the entire artificial intelligence computing power execution framework and processor manufacturers connected to the framework, it is necessary to quickly locate the accuracy problems exposed in the benchmark evaluation and feedback the located problem operators to the operator engineers for maintenance. The multi-level tensor extraction and automated tensor detection strategies proposed in the present invention can quickly reduce the model accuracy problem from the model granularity to the operator granularity, greatly reducing the workload of troubleshooting.

[0078] According to the model accuracy problem locating method proposed in an embodiment of the present invention, the accuracy problems of the target model in the process of executing the target task can be detected step by step according to the multi-level tensor data of the target model in the process of executing the target task, which can avoid the omission of a problem in a certain computing scenario in the process of the target model executing the target task, improve the reliability and comprehensiveness of problem locating, and then locate the position of the accuracy problem in the process of the model executing the target task according to the detection result of the last level, and use the detection result of the current level to assist the detection of the accuracy problem of the next level in the problem locating process, that is, when it is found that the tensor data of a certain level is abnormal, it is possible to directly conduct in-depth analysis on the problem part of that level without conducting a comprehensive and indiscriminate problem investigation on the entire target model, thereby improving the efficiency and accuracy of problem locating.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0080] An embodiment of the present invention also provides a device for locating model accuracy problems.

[0081] like Figure 3 As shown, the model accuracy problem locating device 10 includes: an acquisition module 100, a detection module 200 and a positioning module 300.

[0082] Among them, the acquisition module 100 is used to obtain multi-level tensor data in the process of the target model executing the target task; the detection module 200 is used to perform step-by-step detection of precision problems in the process of the target model executing the target task based on the multi-level tensor data, wherein the detection results of the current level are used to assist in the detection of precision problems at the next level; the positioning module 300 is used to locate the position of the precision problem in the process of the model executing the target task based on the detection results of the last level.

[0083] In an embodiment of the present invention, the detection module 200 is further used to: extract current level tensor data from the multi-level tensor data in the problem processor and the calibration processor respectively; compare the current level tensor data extracted by the problem processor and the calibration processor respectively; and determine the detection result of the current level based on the data comparison results of the problem processor and the calibration processor.

[0084] In an embodiment of the present invention, the detection module 200 is further used to: identify data errors in the data comparison results; if the data error is less than the problem threshold, the detection result of the current level is passed; if the data error is greater than or equal to the problem threshold, the detection result of the current level is failed.

[0085] In the embodiment of the present invention, the detection module 200 is further used to: extract the problem targets that failed the detection in the detection results of the current level; and assist the detection of the next level of accuracy problems based on the problem targets that failed the detection.

[0086] In the embodiment of the present invention, the detection module 200 is further used to: identify the identifier of the problem target that failed the detection; locate the position of the problem target according to the identifier of the problem target; and perform a next level of precision problem detection on the target at the position of the problem target.

[0087] In the embodiment of the present invention, the positioning module 300 is further used to: extract the problem target that failed the detection in the last level detection result; identify the identifier of the problem target that failed the detection; and locate the position of the problem target according to the identifier of the problem target.

[0088] In the embodiment of the present invention, the positioning module 300 is further used to: determine the name and serial number of the problem target according to the identifier of the problem target; and determine the location of the problem target according to the name and serial number of the problem target.

[0089] In an embodiment of the present invention, the acquisition module 100 is further used to: identify the model structure of the target model; divide the target model into multiple levels of targets to be extracted according to the model structure; embed tensor extraction functions before and after the targets to be extracted in the multiple levels of targets to be extracted; and use the tensor extraction function to extract multi-level tensor data of the multiple levels of targets to be extracted.

[0090] In the embodiment of the present invention, the acquisition module 100 is further used to: determine the multi-level data granularity of the target model according to the model structure; and divide the target model into multiple levels of targets to be extracted according to the multi-level data granularity.

[0091] In the embodiment of the present invention, the data granularity of the current level is greater than the data granularity of the next level.

[0092] In an embodiment of the present invention, the acquisition module 100 is further used to: extract tensors of multi-level targets to be extracted using a tensor extraction function; obtain names and serial numbers of multi-level targets to be extracted; and generate multi-level tensor data according to the tensors, names and serial numbers of multi-level targets to be extracted.

[0093] In the embodiment of the present invention, the model accuracy problem location device 10 of the embodiment of the present invention further includes: a generation module.

[0094] Among them, the generation module is used to obtain the data of the problem target in the last level after locating the location of the accuracy problem in the process of the model executing the target task according to the detection results of the last level; and generate an error case based on the data of the problem target.

[0095] It should be noted that, for the description of the features in the embodiment corresponding to the model accuracy problem locating device, reference can be made to the relevant description of the embodiment corresponding to the model accuracy problem locating method, which will not be repeated here.

[0096] An embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, it includes a memory 401 and a processor 402, the memory 401 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above-mentioned model accuracy problem locating method embodiments.

[0097] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned model accuracy problem locating method embodiments when running.

[0098] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0099] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned model accuracy problem locating method embodiments are implemented.

[0100] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] The above is a detailed introduction to a method for locating model accuracy problems provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for locating model accuracy problems, characterized in that: The following steps are involved: Obtain multi-level tensor data during the target model's execution of the target task; According to the multi-level tensor data, the accuracy problems in the process of the target model executing the target task are detected step by step, wherein the detection result of the current level is used to assist the detection of the accuracy problems of the next level; The detection results of the last level are used to locate the location of accuracy problems in the process of the model performing the target task.

2. The method for locating model accuracy problems according to claim 1, characterized in that: The step of detecting the accuracy problem of the target model in the process of executing the target task according to the multi-level tensor data comprises: Extracting current level tensor data from the multiple levels of tensor data respectively in the problem processor and the calibration processor; comparing the current level tensor data extracted by the problem processor and the calibration processor respectively; Based on the data comparison results of the problem processor and the calibration processor, the detection result of the current stage is determined.

3. The method for locating model accuracy problems according to claim 2, characterized in that: The determining the detection result of the current level based on the data comparison result of the problem processor and the calibration processor includes: identifying data errors in the data comparison results; If the data error is less than the problem threshold, the detection result of the current level is detection passed; If the data error is greater than or equal to the problem threshold, the detection result of the current level is detection failure.

4. The method for locating model accuracy problems according to claim 1, characterized in that: The method of using the detection result of the current level to assist the detection of the accuracy problem of the next level includes: Extracting the problem targets that failed the detection in the detection results of the current level; Assist the detection of the next level of accuracy problems based on the problem targets that failed the detection.

5. The method for locating model accuracy problems according to claim 4, characterized in that: The step of assisting the detection of the next level of precision problems according to the problem target that failed the detection includes: Identify the identification of the problem target that failed the detection; Locating the position of the problem target according to the identifier of the problem target; The target at the position of the problem target is detected for the next level of precision problem.

6. The method for locating model accuracy problems according to claim 1, characterized in that: The location of the accuracy problem in the process of locating the model to perform the target task according to the detection result of the last level includes: Extracting the problem targets that failed the detection in the detection results of the last level; Identify the identification of the problem target that failed the detection; The position of the problem target is located according to the identifier of the problem target.

7. The method for locating model accuracy problems according to any one of claims 5 to 6, characterized in that: The step of locating the position of the problem target according to the identifier of the problem target includes: Determining the name and serial number of the problem target according to the identifier of the problem target; The location of the problem object is determined according to the name and sequence number of the problem object.

8. The method for locating model accuracy problems according to claim 1, characterized in that: The step of obtaining multi-level tensor data during the process of the target model executing the target task includes: identifying a model structure of the target model; Dividing the target model into multiple levels of targets to be extracted according to the model structure; Embedding a tensor extraction function before and after the target to be extracted in the multi-level targets to be extracted; The tensor extraction function is used to extract multi-level tensor data of the multi-level targets to be extracted.

9. The method for locating model accuracy problems according to claim 8, characterized in that: The step of dividing the target model into multiple levels of targets to be extracted according to the model structure includes: Determining the multi-level data granularity of the target model according to the model structure; The target model is divided into multiple levels of targets to be extracted according to the multiple levels of data granularity.

10. The method for locating model accuracy problems according to claim 9, characterized in that: The data granularity of the current level is larger than the data granularity of the next level.

11. The method for locating model accuracy problems according to claim 8, characterized in that: The step of extracting multi-level tensor data of the multi-level target to be extracted by using the tensor extraction function includes: Extracting the tensors of the multi-level targets to be extracted using the tensor extraction function; Obtain the names and serial numbers of the multi-level targets to be extracted; Generate multi-level tensor data according to the tensors, names and serial numbers of the multi-level targets to be extracted.

12. The method for locating model accuracy problems according to claim 8, characterized in that: After locating the location of the accuracy problem in the model's execution of the target task based on the detection results of the last level, it also includes: Obtaining data of the problem target in the last level; Generate error cases based on the data of the problem target.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the method for locating model accuracy problems as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for locating model accuracy problems as described in any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for locating model accuracy problems as described in any one of claims 1 to 12 are implemented.

Images